Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

PROCERT issues

1,897 views
Skip to first unread message

Gervase Markham

unread,
Sep 7, 2017, 11:18:27 AM9/7/17
to mozilla-dev-s...@lists.mozilla.org
Mozilla has decided that there is sufficient concern about the
activities and operations of the CA "PROCERT" to collect together our
list of current concerns. That list can be found here:
https://wiki.mozilla.org/CA:PROCERT_Issues

Note that this list may expand or reduce over time as issues are
investigated further, with information either from our or our
community's investigations or from PROCERT.

We expect PROCERT to engage in a public discussion of these issues and
give their comments and viewpoint. We also hope that our community will
make comments, and perhaps provide additional information based on their
own investigations.

When commenting on these issues, please clearly state which issue you
are addressing on each occasion. The issues have been given identifying
letters to help with this.

At the end of a public discussion period between Mozilla, our community
and PROCERT, which we hope will be no longer than a couple of weeks,
Mozilla will move to make a decision about the continued trust of
PROCERT, based on the picture which has then emerged.

Gerv

Ryan Sleevi

unread,
Sep 7, 2017, 5:27:55 PM9/7/17
to Gervase Markham, mozilla-dev-security-policy
(Unless stated, wearing a personal hat)

Hi Gerv,

Do you have an anticipated time period for discussion? That is, what
represents a time for which PROCERT may submit feedback to have it
considered, and at what point you will consider discussion closed?

Based on the information provided, Issue K represents an unconditional
security issue, in as much as names such as "autodiscover" and "owaserver"
are widely-used domains for Outlook Web Access. Many clients attempt to
access resources at this (unqualified) domain, relying on the combination
of DNS suffix search and locally-trusted certificates to ensure proper
resolution. By issuing a publicly trusted certificate for this name - and
then failing to revoke it - represents a critical security risk and
arguably a dereliction of responsibility.

Combined with Issue D and Issue G, it is questionable as to how it was ever
validated, and suggest serious failings over the most critical security
control of a CA - which is validation of a domain.

Combined with Issue L, Issue Q, Issue R, Issue X, and Issue W, serious
questions are raised about the oversight and technical ability of the
staff, as these are indicative of serious control failures.

Outside of Issue K, I would suggest that Issue O and Issue S show a lack of
awareness of developments in the CA ecosystem, as both of these controls
were direct responses to widely reported CA security issues. The failure to
take appropriate steps - or to appreciate the reasons behind such steps -
are indicative of a systematic misunderstanding of the security function of
a CA.

On the basis of the sum of these issues, it would seem that the criteria in
Section 7.3 of Mozilla policy -
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/
- is met: "Mozilla will disable or remove a certificate if the CA
demonstrates ongoing or egregious practices that do not maintain the
expected level of service or that do not comply with the requirements of
this policy."

Alex Gaynor

unread,
Sep 8, 2017, 9:15:21 AM9/8/17
to Ryan Sleevi, mozilla-dev-security-policy, Gervase Markham
I believe it's important to consider more than just the issues themselves,
and to look at a CA's response to the issues. In the past weeks, we've done
a lot of really fantastic work to push CAs on publishing more comprehensive
post-mortems, and several CAs have distinguished themselves with timely
responses and solid analysis of the underlying causes of failures.

In that frame, I want to highlight a few issues that raise serious concerns
to me:

* In responding to issue D, PROCERT displayed a failure to analyze and
understand the issues.
* In responding to issue O, PROCERT once again claimed there wasn't a
problem after evidence has already been provided of the issue. Further,
they appear to have significant challenges maintaining and updating their
PKI software.
* In responding to issue E, PROCERT made several rounds of incorrect
statements about their certificate issuance; they have not substantively
responded to the issue.

I believe that PROCERT's inability to respond to problem reports in a
timely and correct fashion, even more so than the certificate issuance
itself, represents an ongoing practice practice which is not in keeping
with the goals of the Mozilla Root Program.

Alex
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Gervase Markham

unread,
Sep 8, 2017, 10:55:41 AM9/8/17
to ry...@sleevi.com
On 07/09/17 22:27, Ryan Sleevi wrote:
> Do you have an anticipated time period for discussion? That is, what
> represents a time for which PROCERT may submit feedback to have it
> considered, and at what point you will consider discussion closed?

I don't want to place a hard limit on it because often finding the truth
requires several rounds of interaction. But, as noted in the original
post, I would expect to be moving towards a determination of a response
in a matter of weeks (rather than months).

Gerv

Jakob Bohm

unread,
Sep 8, 2017, 2:39:46 PM9/8/17
to mozilla-dev-s...@lists.mozilla.org
Although violating the same rules, and involving the same certificates;
for purposes of risk assessment I think issue K should be divided into
two issues:

K1 (most serious): Multiple certificates issued for the bare hostname
OWASERVER (uppercase, no qualifying domain). As pointed out by Ryan
Sleevi, many e-mail clients (including mobile clients) may look for
this name on their local DNS search list and may or may not (depending
on client implementation) accept any of these bare certificates as
proving they are talking to their "home" mail server. So far none of
the other MS mail server magic/default names have been found as bare
name SANs.

K2 (very serious): Multiple certificates issued to potentially
non-unique subdomains of the local. TLD previously common for LAN DNS,
but now mostly reserved for multicast DNS. These violations should
only pose a risk to clients who have somehow chosen the same arbitrary
2. level domain under local. as the certificate holder(s). The main
issue here is that since the local. TLD doesn't have an official
registry, there is no way that the CA could have properly validated
that *any* applicant was the proper owner of such a 2nd level domain,
because noone is.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Ryan Sleevi

unread,
Sep 8, 2017, 2:42:42 PM9/8/17
to Jakob Bohm, mozilla-dev-security-policy
On Fri, Sep 8, 2017 at 2:39 PM, Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 07/09/2017 17:17, Gervase Markham wrote:
>
> Although violating the same rules, and involving the same certificates;
> for purposes of risk assessment I think issue K should be divided into
> two issues:
>

Note, I was explicitly suggesting we not do this, because this introduces a
greater level of subjectivity of assessment, and based on incomplete or
unknowable information. For this reason, ensuring a consistent application
of risk (e.g. the factors that allowed this to happen are the same) is far
more beneficial for the community and for consistency in application of
policy.

So I do not believe we should split these issues up, and do not think it
would help the discussions.

PSC Procert

unread,
Sep 9, 2017, 6:09:35 AM9/9/17
to mozilla-dev-s...@lists.mozilla.org
Good Afertnoon

In order to answer the points of the wiki, we make the following explanations



Issue D: URI in CN and dnsName SAN (December 2016)

Procert:

Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process. We completed those actions in this point.


Issue E: Issuance of 1024-bit Certificate (December 2016)
Procert:

Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate in 2048, supporting into the installation process. We completed those actions in this point.


Issue G: Internal IP Address in SAN (March 2015 - March 2017)

Procert:

Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process. We completed those actions in this point.



Issue I: CN Not Also In SAN (March 2016 - June 2017)

Procert:

Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process. We completed those actions in this point.



Issue J: Use of keyCertSign in Leaf Certificates (October 2016 - June 2017)

Procert:

The template for this certificate was fixed and based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process. We completed those actions in this point.



Issue K: Internal DNS Names in Certificates (May - June 2017)

Procert:

Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process. We completed those actions in this point.



Issue L: helloburgershack.com (June - July 2017)

Procert:

Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process. This client provides a production window after the next Tuesday in order to proceed with the revoke and the reissue of the certificate. Pending action.



Issue M: CPS not in RFC 3647 or RFC 2527 Format (2011 - August 2017)

Procert:

After a validation process, we modify the RFC number in our documentation. We complete this point.



Issue N: otherNames in Certificate SAN (2011 - August 2017)

Procert:
Based on internal testing and validation, we contact the customer, request a window to generate a new CSR and review and reissue the certificate, we will also support the installation process. We keep in touch with customers with active SSL to proceed with this point. We advance as much as possible with customers. Some of these certificates have expired. For the issue of new certificates, verify the observations contained in the number V


Issue O: OCSP Servers Return "Good" for Non-Existent Certificates (Unknown - August 2017)

Procert:

As we explained into Mozilla Bug (1391058), when we applied the Microsoft tool, the system shows this message “In the Value data box, type the path to the directory you created in step 3 of the directory structure procedure and that contains the issued serial numbers, and then click OK.”.

We refresh or restart the service, then, the OSCP registry is automatically deleted. For testing we use different versions of Windows Server (2008, 2012 and 2016) all the versions present the same result. Additionally we ask for an answer at Microsoft TechNet please https://social.technet.microsoft.com/Forums/windowsserver/es-ES/981f6e48-dc25-4eeb-a1d6-0bc72b9b4fc9/ocsp-online-responder-service-assume-a-certificate-that-is-not-included-in-the-crl-as-a-valid-and?forum=winserversecurity

Now we stay contacting Microsoft in order to obtain and adequate procedure or batch. In paralleled we work in our own OCSP software.



Issue P: Use of SHA-1 To Sign OCSP Responses (Unknown - August 2017)

Procert:

We check the standard, the OCSP certificate is SHA 256, the answer in this case is an observation. We work to check and validate the adjust to SHA 256 in the OCSP answer. This situation does not contravene any standard.



Issue Q: CRL Distribution Points Using HTTPS (August 2012 - August 2017)

Procert:

Please validate the certificate issuance. This certificate is not issued by PSC PROCERT



Issue R: Incorrect Encoding of or Inappropriate Use of TeletexString (December 2015 - August 2017)

Based on internal testing and validation, we contact the customer, request a window to generate a new CSR and review and reissue the certificate, we will also support the installation process. We keep in touch with customers with active certificate to proceed with this point. We advance as much as possible with customers.



Issue S: Non-Random Serial Numbers (September 2016 - August 2017)

Procert:

We check the observation. Procert technical staff applied the observation and fix the system in this particular point.



Issue T: Inappropriate Key Usage Value of "Key Agreement" (October 2016 - August 2017)

The template for this certificate was fixed and based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process.



Issue V: Failure to Respond Quickly To Problem Reports (August 2017)

Procert:
We tested our certificates in our test environment, we analyze the issues and apply measure to solve the issues, later, the team takes actions, modify certificates templates, work in the CA, made test, generated new certificates in our internal environment. Then contacted the clients and agree a revocation date, revoke all the certificates with problem and reissue the certificates with the standard complying, check the correct application of CA Browser Forum, implant a regular training program (including test (operational and theory) to our staff in order to prevent and solve any issue, finally proceed with a dismissal of one operator.



Issue W: Test Certificates Issued in Publicly Trusted Hierarchies (August 2017)

Procert:

The certificate was revoked and we put enforce our security policy in order to prevent futures events. Also we instruct our operational staff regarding internal process and IT Security topics



Issue X: High Percentage of Revocations (August 2017)

Procert:

Staff takes actions considering the observations includes into the Mozilla bug. In order to that, we made an internal audit, check the issuance process, detected al the unconformities, contacting the clients, agree with the clients a windows to revoke and installing the new certificates. In order to duly comply, we proceeded to revoke certificates with observations (Mozilla Bug). We planning replace all the certificates with observations. That why you can appreciate a high percent into the certificates revocations


Note:

A deadline has been set until the 14th of this month to revoke and re-issue certificates with problems

Gervase Markham

unread,
Sep 11, 2017, 7:03:52 AM9/11/17
to PSC Procert
Hi Alejandro,

Thank you for this initial response. It is, however, far less detailed
than we would like to see. In the email I sent to you letting you know
that we were looking at PROCERT, I wrote:

"You may wish to review a similar issue list we created for Symantec:
https://wiki.mozilla.org/CA:Symantec_Issues
in order to see how such a wiki page might develop in the future. For
example, we would expect most or all issues on your list to soon carry a
"PROCERT Response" section. Good responses will be more than "yes, we
have revoked the certificate" or even "we have revoked, and updated our
software". We want to know how each problem occurred in the first place,
and why it was not detected until now by PROCERT's existing systems for
quality control. This wiki page:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance
gives best practices in responding to incidents and writing reports,
which may prove useful in formulating your responses."

It seems you have chosen not to follow this advice, as many of your
responses consist solely of an assertion that you have revoked and
replaced the problematic certificates. So, because there is no root
cause analysis, we have no assurance at all that the issue will not
occur again.

Even beyond that major concern, I think there are some of these issues
where you have not entirely understood the problem.

On 08/09/17 20:09, PSC Procert wrote:
> Issue E: Issuance of 1024-bit Certificate (December 2016)
> Procert:
>
> Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate in 2048, supporting into the installation process. We completed those actions in this point.

Your cut-and-paste answer is inapplicable in this case, because the
"client" was PROCERT itself - the certificate in question was an OCSP
responder cert.

> Issue L: helloburgershack.com (June - July 2017)
>
> Procert:
>
> Based on internals test and validation, we contacting the client, we asking for a new CSR and proceed to revoke and reissue the certificate, supporting into the installation process. This client provides a production window after the next Tuesday in order to proceed with the revoke and the reissue of the certificate. Pending action.

What was sought here was an explanation of why it took five attempts to
issue this certificate, and even the last version had problems.

> Issue M: CPS not in RFC 3647 or RFC 2527 Format (2011 - August 2017)
>
> Procert:
>
> After a validation process, we modify the RFC number in our documentation. We complete this point.

I think you misunderstand. The issue here is not that some RFC number in
the document is wrong, but that your document is not arranged according
to the layout given in RFC3647. To fix this would require completely
rearranging the document. Are you saying that you have already completed
this? If so, can we see a copy of the updated document?

> Issue O: OCSP Servers Return "Good" for Non-Existent Certificates (Unknown - August 2017)
....
> Now we stay contacting Microsoft in order to obtain and adequate procedure or batch. In paralleled we work in our own OCSP software.

I suggest that writing and deploying your own OCSP software is unlikely
to be a route towards speedy conformance with all applicable
regulations. If you are unable to get Microsoft's software to function,
I would suggest approaching another vendor. But perhaps another CA has
the Microsoft software working, and could help out? You may want to ask
on the CAB Forum list.

> Issue P: Use of SHA-1 To Sign OCSP Responses (Unknown - August 2017)
>
> Procert:
>
> We check the standard, the OCSP certificate is SHA 256, the answer in this case is an observation. We work to check and validate the adjust to SHA 256 in the OCSP answer. This situation does not contravene any standard.

No, but it does contradict your assertions in your responses to two
previous Mozilla CA Communications.

> Issue Q: CRL Distribution Points Using HTTPS (August 2012 - August 2017)
>
> Procert:
>
> Please validate the certificate issuance. This certificate is not issued by PSC PROCERT

You are quite right - my apologies. It's from a different part of the
Venezuelan Goverment's hierarchy. I was sent a large list of
certificates with this problem but a few others I've sampled also seem
to be from other parts. I will look into this and see if I can find an
example issued by PROCERT; if not, I will strike this issue.

> Issue S: Non-Random Serial Numbers (September 2016 - August 2017)
>
> Procert:
>
> We check the observation. Procert technical staff applied the observation and fix the system in this particular point.

Please can you explain the method of construction used for the most
recent serial numbers, e.g. the one on https://crt.sh/?id=207894453?

> Issue X: High Percentage of Revocations (August 2017)
>
> Procert:
>
> Staff takes actions considering the observations includes into the Mozilla bug. In order to that, we made an internal audit, check the issuance process, detected al the unconformities, contacting the clients, agree with the clients a windows to revoke and installing the new certificates. In order to duly comply, we proceeded to revoke certificates with observations (Mozilla Bug). We planning replace all the certificates with observations. That why you can appreciate a high percent into the certificates revocations

The issue being raised here is not the large number of certificates that
PROCERT is revoking (we might expect a large number of revocations,
given all of the problems), but the fact that a large number are revoked
very shortly after they are issued. Why does that keep happening? If
they are being issued incorrectly, why are corrective steps not being taken?

Thank you in advance for your timely further responses.

Gerv

Gervase Markham

unread,
Sep 18, 2017, 9:27:18 AM9/18/17
to PSC Procert
On 11/09/17 12:03, Gervase Markham wrote:
> Thank you for this initial response. It is, however, far less detailed
> than we would like to see.

I have not had any further updates from PROCERT. I have tried to reflect
their responses from this email here:
https://wiki.mozilla.org/CA:PROCERT_Issues

We hope to conclude the discussion at the end of this week, although I
am having minor surgery on Wednesday, so it may be next week.

Gerv

alejand...@gmail.com

unread,
Sep 21, 2017, 5:08:37 PM9/21/17
to mozilla-dev-s...@lists.mozilla.org
Dear Gerv, I have attached a document that gives us a greater response to each of the points, as well as Mr. Oscar Lovera sent you an email with the same information

https://www.dropbox.com/s/qowngzzvg5q5pjj/Mozilla%20issues.docx?dl=0

Regards

Patrick Figel

unread,
Sep 21, 2017, 5:43:13 PM9/21/17
to dev-secur...@lists.mozilla.org
On 21/09/2017 23:08, alejandrovolcan--- via dev-security-policy wrote:
> Dear Gerv, I have attached a document that gives us a greater
> response to each of the points, as well as Mr. Oscar Lovera sent you
> an email with the same information
>
> https://www.dropbox.com/s/qowngzzvg5q5pjj/Mozilla%20issues.docx?dl=0


To save everyone else some time trying to find out what has changed,
these are the additions for every issue, diffed against the original
response[1]:

------------------------------------------------------------------------

Issue D:

PROCERT initially claimed that this was entirely RFC-compliant, a claim
comprehensively rebutted by Ryan Sleevi.

This certificate was issued with the modifications and here it is
possible to appreciate the evidence https://crt.sh/?id=209727657

------------------------------------------------------------------------

Issue E:

This certificate is not in use and is appended to the internal
revocation schedule, the current certificate is 2048 in length and can
be seen in the attached OCSP response (ocsp.txt)

------------------------------------------------------------------------

Issue G:

Please find evidence of new certificate issued without problems
https://crt.sh/?id=202869851

------------------------------------------------------------------------

Issue I:

These certificates were revoked and generated again, please consult
through the following link
https://crt.sh/?iCAID=750&minNotBefore=2000-01-01&n=1000&cablint=25

------------------------------------------------------------------------

Issue J:

Although certificates are not certified, the certificates have been
revoked and corrected, please check the link
https://crt.sh/?iCAID=750&minNotBefore=2000-01-01&cablint=782

------------------------------------------------------------------------

Issue K:

Corrective action taken, please consult the link
https://crt.sh/?id=197158298&opt=cablint

------------------------------------------------------------------------

Issue L:

That certificate was revoked and generated again, please consult through
the following link https://crt.sh/?id=167929373

------------------------------------------------------------------------

Issue M:

This is not an infringement or violation of the RFC. Regarding to the
language, the national language in Venezuela is the Spanish. We will
work to updated the English version of this document and posted in the
web page.
https://www.procert.net.ve/documentos/AC-D-0011.pdf
https://www.procert.net.ve/documentos/AC-D-0003.pdf

Standard
https://tools.ietf.org/html/rfc3647
https://tools.ietf.org/html/rfc2527

------------------------------------------------------------------------

Issue N:

Taking into account what the Baseline says, it states the following:
i. Other Subject Attributes
All other optional attributes, when present within the subject field,
MUST contain information that has been verified by the CA.

Indicates that another field can be included as long as it is
information verified by the CA in this case the CA verifies the number
of RIF of each company and also the field is with an OID for that
purpose, which is 2.16.862.2.2

------------------------------------------------------------------------

Issue O:

Our OCSP works without any problem in the validation with automatic
tools such as certutil, even with the same command openssl, in
consultation under standard.

The standard openssl query is done in the following way: openssl ocsp
-issuer issuer.cer -cert cert.cer -url http://ura.procert.net.ve/ocsp
-noverify -text.

We detected that OCSP does work correctly with the Microsoft tool.

Open SSL creates a problem when doing a non-standard query (hacking). We
are already working the point with Microsoft and we even have an
assigned ticket.

------------------------------------------------------------------------

Issue P:

No RFC 2560 nor RFC 5280 is being violated.

------------------------------------------------------------------------

Issue R:

The certificate was revoked and reissued, please see the link
https://crt.sh/?id=194225991

------------------------------------------------------------------------

Issue S:

Already it was given previous answer to this point and was solved
modifying certain parameters in our CA, which counts on a CSPRNG that
acts like generator of serial numbers and was modified in its register
so that it increased the capacity and strength of the serial numbers
that generates automatically and without human intervention.

------------------------------------------------------------------------

Issue T:

These certificates were revoked and generated again, please consult
through the following link
https://crt.sh/?iCAID=750&minNotBefore=2000-01-01&n=1000&cablint=734
Additionally, we have already modified the certificate template to
prevent it from containing this key usage.

------------------------------------------------------------------------

Issue V:

No additions.

------------------------------------------------------------------------

Issue W:

This action does not infringe any standard, in any case by security
policy should not be issued for a long time, since despite having a
quality environment, you can always test in the production environment
some test certificate, all the certificates were revoked and were only
issued under an authorization, since for the issuance of certificate
there is a mechanism of approval of 3 levels, In addition the
certificates were revoked
https://crt.sh/?id=197073488&opt=cablint
https://crt.sh/?id=197155020&opt=cablint

------------------------------------------------------------------------

Issue X:

No additions.

------------------------------------------------------------------------

Apologies in case I missed anything, the formatting of the document was
not particularly consistent.


[1]:
https://groups.google.com/d/msg/mozilla.dev.security.policy/lqZersN26VA/nrXoE9JiAQAJ

Daniel Cater

unread,
Sep 23, 2017, 10:23:02 AM9/23/17
to mozilla-dev-s...@lists.mozilla.org
Thank you for creating the diff and posting it here.

Procert's continued statements of "behaviour x does not violate the RFC" / "behaviour x does not infringe the standard" show that they do not recognise the Baseline Requirements as something that needs to be adhered to in order to remain in the browser root programs.

This is a total misunderstanding / ignorance of why the BRs exist, and of how the browser root programs work, which is very worrying.

alejand...@gmail.com

unread,
Sep 26, 2017, 4:56:36 PM9/26/17
to mozilla-dev-s...@lists.mozilla.org
In the following link you can find the CPS in English language

https://www.procert.net.ve/documentos/CPS-PROCERT.pdf

uri...@gmail.com

unread,
Sep 27, 2017, 12:37:55 AM9/27/17
to mozilla-dev-s...@lists.mozilla.org
Why does the document say "Date: 11/07/17" on every page, and the signed pdf metadata say

<xmp:CreateDate>2017-09-25T17:14:35-04:00</xmp:CreateDate>
<xmp:ModifyDate>2017-09-25T17:18:07-04:00</xmp:ModifyDate>
<xmp:MetadataDate>2017-09-25T17:18:07-04:00</xmp:MetadataDate>

Kathleen Wilson

unread,
Sep 27, 2017, 12:56:27 PM9/27/17
to mozilla-dev-s...@lists.mozilla.org
In past incidents, we have provided a list of action items that the CA must complete before they can be re-included in Mozilla's root store.

What action items do you all think PROCERT should complete before they can be re-included in Mozilla's root store?

What do you think should happen if PROCERT completes those action items before their PSCProcert root is actually removed?

Thanks,
Kathleen



Matthew Hardeman

unread,
Sep 27, 2017, 1:54:45 PM9/27/17
to mozilla-dev-s...@lists.mozilla.org
On Wednesday, September 27, 2017 at 11:56:27 AM UTC-5, Kathleen Wilson wrote:

> What action items do you all think PROCERT should complete before they can be re-included in Mozilla's root store?

> What do you think should happen if PROCERT completes those action items before their PSCProcert root is actually removed?

This is at least the second recent instance in recent memory where technical competencies and/or other issues cause a CA to respond in a less than truthful way. (Startcom, also, but in the original Startcom case it was also alleged that there was intentional deception.)

While I think it would be a stretch to suggest that any of PROCERT's answers to questions posed to them were deceptive by intention, there were certainly statements of fact asserted by PROCERT in their answers which were factually incorrect and thus not truthful. (Particularly responses alleging compliance with certain RFCs while this was observably not the case. I imagine but don't recall with specificity that there are other examples).

There also seemed a reluctance to provide real answers to the questions asked by the community.

While there was also technical cause to eject these program participants, it looks / feels like the community's object to these CAs in the end is really about lack of confidence in their responses and willingness to respond. There seemed to be a real reluctance (to put it mildly) in these programs to comply with evolving industry requirements.

StartCom's recent attempts to get re-added to the program certainly demonstrated some flaws (which StartCom claims they now have a handle on) in their stand-up of their new PKI.

In the case of StartCom, I can not help but feel that they are being held to an especially high standard (higher than other prior adds to the program) in this new PKI because of who they are -- despite the fact that management and day-to-day decisions are a completely different team.

Where I am headed with this is a concern that perhaps no amount of technical remediation can really get these entities back in the graces of the community.

Honestly, that follows pretty logically. No technical mitigation on earth is an appropriate remedy for any of delays, boilerplate verbiage, or factual inaccuracies in responses to community and program inquiries.

To build a remediation plan comprised of technological steps and requirements stops far short of actually establishing faith in competency and trust of compliance.

I believe this is reflected in the StartCom re-inclusion request. Certainly the level of scrutiny applied is heightened -- and this is not wrong.

On the other hand, if you move to remedies such as sweeping changes in management being required, there is the chance that you diminish quality and experience of the technical management. There is a chance of improvement also, but it is not a guarantee.

The trouble I see is that re-establishing trust in either of these entities pragmatically would require a significant technology remediation as well as serious management remediation.

But, looking at the experience StartCom has had -- if you, Kathleen, were counsel to StartCom or PROCERT, wouldn't you be saying something along the lines of "We need to talk about corporate structure, entity name, etc. This is a whole new management, as required by the program. The number of technological steps asked are significant and tedious. It's a new house anyway, why don't we name it as such, spin up a new corporate entity, stand up a new PKI to best practices under custody and management of the new executive team and apply to the root program as a new and novel entity -- which you are. Let's not bring any unnecessary baggage with the old name."?

I'm not sure the issue has public arisen, but I am guessing that Mozilla and the other root programs DO concern themselves with beneficial ownership of the various CAs but at the same time, absent evidence of undue influence / coercion by beneficial owners, broadly regard the management and operations teams of the CA applicant / participant as the key individuals for which trust must be vested.

Among the reasons I believe that is there are numerous private equity and/or publicly traded entities which wholly own publicly trusted CAs. There must be _some_ actual felons with not insignificant ownership interests in those entities. The executive management and operations teams are likely beyond reproach, however.

In summary, it is difficult for me to see how either of PROCERT or StartCom would not be better served to start over for real, with trustworthy management on all new infrastructure, beneficial ownership be damned because I'm not sure you can or would or should care about that end.

It seems to me that any remediation plan which comprises a cross-section of both technical issues AND management competencies/trustworthiness that we arrive at a plan which is more onerous than just starting over. Thus, why not just start over?

I understand that this ultimately means stigmatizing certain individual human beings as "unfit to run / work at a publicly trusted CA". But that already happens anyway. (If not by the root programs, at least by the community.) Why not admit that and formalize it in a transparent way?

Thanks,

Matt Hardeman

okaphone.e...@gmail.com

unread,
Sep 27, 2017, 4:21:44 PM9/27/17
to mozilla-dev-s...@lists.mozilla.org
This it about trust. No more, no less. Once you've lost trust, what can be done to restore it really depends on how you've lost it. Jumping through a series of Mozilla defined (technical) hoops is never going to be convincing. Especially not if it takes more several tries. ;-)

So, was it incompetence? Then show that the lacking skills have be acquired. Was it human error? Show that you are not relying on human accuracy anymore. Etcetera...

And it makes definitely most sense parties who loose trust come up with a relevant plan for this themselves. That is, after identifying what the problem was themselves. They will unfortunately have to do that transparently. Here where everybody can see it. And where questions can be asked about it and answered.

This will definitely be a lot of work and it will certainly not be easy. But I'd say that anything less is too little to result in regaining trust.

(Apart from this I personally don't think any CAs will/should be able to find a way back from loosing trust through willfully lying.)

CU Hans

Gervase Markham

unread,
Sep 28, 2017, 12:51:13 PM9/28/17
to Matthew Hardeman
On 27/09/17 18:54, Matthew Hardeman wrote:
> In the case of StartCom, I can not help but feel that they are being
> held to an especially high standard (higher than other prior adds to
> the program) in this new PKI because of who they are -- despite the
> fact that management and day-to-day decisions are a completely
> different team.
>
> Where I am headed with this is a concern that perhaps no amount of
> technical remediation can really get these entities back in the
> graces of the community.

I don't know if it's quite as absolute as that, but recent incidents
have caused me to ponder somewhat on the nature of trust. The root
program is all about trust, and trust is not something which can be
encoded in audits, checkboxes and rules. This will always be a tension
at the heart of our root program - we are trying to be as objective as
we can about something which is ultimately subjective.

The nature of trust is that it's harder to regain than it is to gain in
the first place. Just ask someone who's been the victim of adultery - or
someone who is a now-repentant adulterer. Rightly or wrongly, people get
a first chance, but it's tough to get a second. I think you are right
when you conclude that this is just the way of things, and we should
accept it rather than kick against it.

Gerv

Matthew Hardeman

unread,
Sep 28, 2017, 5:51:40 PM9/28/17
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
On Thu, Sep 28, 2017 at 11:50 AM, Gervase Markham <ge...@mozilla.org> wrote:

>
> The nature of trust is that it's harder to regain than it is to gain in
> the first place. Just ask someone who's been the victim of adultery - or
> someone who is a now-repentant adulterer. Rightly or wrongly, people get
> a first chance, but it's tough to get a second. I think you are right
> when you conclude that this is just the way of things, and we should
> accept it rather than kick against it.
>
> Gerv
>

I am curious how accepting that reality might impact the program.

It raises several questions in my mind:

Does the root program formally recognize the trust required of key
executives and operations staff?

Does the program actually concern itself with beneficial ownership -
particularly when all parties are asserting that the ownership is passive?
If so, how are those assertions and determinations vetted and documented?

Regardless of ownership, does the root program require any specific formal
assertions or affirmations by CA's key executive staff? Do those or would
those include declarations such as "I am formally declaring that I was/will
be the senior executive responsible for management and operational
decisions of the CA for the period ___. "?

Would / should the program require that trusted individuals in key roles
regularly certify such claims of responsibility and require that these
individuals personally acknowledge an affirmative responsibility to notify
the program if their role transitions or that the exercise of their
professional discretion within their role has been subverted or curtailed?

The spirit of these questions is to inspire some thought as to who are the
individuals in whom trust must be granted are and what precisely we are
trusting them with and how can we capture in a more formal way just what it
is that they are being trusted with and what the program's and community's
expectations are regarding any formal affirmative responsibilities related
to the trust that's being granted.

Business entities lack thought and discretion. It's the trusted people
within a CA who either operate the CA pursuant to standard or do not. How
can those specific extensions of trust be formally documented and what
obligations does that impose on those individuals. How can that be
captured in such a way as to hold those who've been granted trust
accountable?

By way of example, I suspect that a certain R.W. is effectively subject to
a lifetime ban from executive and trusted operations roles in any CA in the
Mozilla root program. I also suspect that's not really formally documented
as such in the general case. I can not imagine that anyone in the
management of the Mozilla root program would respond with a "Erm, yes, I
guess" to a CA asking the hypothetical question "We're contemplating hiring
Richard Wang for COO of our currently publicly trusted CA. But we're good,
though, right?" To be quite clear, I am not asking about any particular
individual, but making a point that, as you said, trust is far harder to
regain than it is to be granted initially.

Formalizing the program's recognition of the key participants of the
various roles offers the opportunity for holding individual bad actors
accountable while also giving the opportunity to hold harmless an executive
who reports an attempt at a silent, internal coercion as to his autonomy in
his role (perhaps by a purportedly arms-length owner) and that said
executive is abandoning the role officially and wishes not to be held
responsible for events at said CA beyond time point X.

Formalizing the responsibilities and liabilities of the individuals allows
punishment and reward. It also has the potential for providing a mechanism
for redress, where warranted. Generally speaking it is difficult for a
person who is unofficially banned to get officially unbanned. It follows
that it's hard to get a status changed when others won't acknowledge the
status.

Just some thoughts...

Matt

okaphone.e...@gmail.com

unread,
Sep 29, 2017, 10:50:24 AM9/29/17
to mozilla-dev-s...@lists.mozilla.org
I'd say this implies two things.

First CAs should be wary of the possibility loosing trust. For reacting/responding timely and adequately to any concerns being raised, instead of ignoring them or waiting to "see how they develop", is a lot easier than any alternative, I'd say.

The other thing is that it is makes sense that CAs (or people) who have lost trust will have to figure out themselves how to get back to trust. Like I said, it depends a lot on how exactly trust was lost in te first place.

And bottom line it is their problem, not Mozilla's. They created it, it is not reasonable to expect/ask a root program to prescribe how to do that.

CU Hans

Eric Mill

unread,
Sep 29, 2017, 5:52:49 PM9/29/17
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Matthew Hardeman
On Thu, Sep 28, 2017 at 12:50 PM, Gervase Markham via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 27/09/17 18:54, Matthew Hardeman wrote:
> > In the case of StartCom, I can not help but feel that they are being
> > held to an especially high standard (higher than other prior adds to
> > the program) in this new PKI because of who they are -- despite the
> > fact that management and day-to-day decisions are a completely
> > different team.
> >
> > Where I am headed with this is a concern that perhaps no amount of
> > technical remediation can really get these entities back in the
> > graces of the community.
>
> I don't know if it's quite as absolute as that, but recent incidents
> have caused me to ponder somewhat on the nature of trust. The root
> program is all about trust, and trust is not something which can be
> encoded in audits, checkboxes and rules. This will always be a tension
> at the heart of our root program - we are trying to be as objective as
> we can about something which is ultimately subjective.
>
> The nature of trust is that it's harder to regain than it is to gain in
> the first place. Just ask someone who's been the victim of adultery - or
> someone who is a now-repentant adulterer. Rightly or wrongly, people get
> a first chance, but it's tough to get a second. I think you are right
> when you conclude that this is just the way of things, and we should
> accept it rather than kick against it.
>

That dynamic is natural, but accepting that this dynamic exists is
different than giving into it in some absolute way. When offering second
chances, requiring that the person/org fulfill certain conditions that
speak directly to their ability to have learned and adapted from the thing
they failed at the first time is an approach that accepts this dynamic,
without shutting the door on people or organizations that have grown as a
result of the experience.

I think it would arguably lead to worse behavior, and less disclosure of
incidents and mistakes, if Mozilla adopted a posture where second chances
are rarely given. Not saying that's what's being said here, but I think
it's worth emphasizing that the first principle here should be to optimize
for incentivizing the behavior you want out of the CA community that
protects users and increases information sharing.

-- Eric


>
> Gerv
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



--
konklone.com | @konklone <https://twitter.com/konklone>

alejand...@gmail.com

unread,
Oct 2, 2017, 11:10:58 AM10/2/17
to mozilla-dev-s...@lists.mozilla.org
Dear Mozilla CA Root Team,

After reviewing Mr. Gervase's reply, referring to the exclusion of the PSC PROCERT from the Mozilla trust repository and having seen the antecedents existing in multiple previous cases, it is evident that in all cases it was offered through the bug of a mechanism of remediation and the ACs were adequately informed about the open observations and even in some cases are closed with simple statements about how the case is remedied.

The technical aspects indicated in the bug and its answers are included below:

1. Serial Number does not meet the standard.
RFC 5280, in section 4.1.2.2 states the following:

“4.1.2.2. Serial Number
The serial number MUST be a positive integer assigned by the CA to each certificate. It MUST be unique for each certificate issued by a given CA (i.e., the issuer name and serial number identify a unique certificate). CAs MUST force the serialNumber to be a non-negative integer.
Given the uniqueness requirements above, serial numbers can be expected to contain long integers. Certificate users MUST be able to handle serialNumber values up to 20 octets. Conforming CAs MUST NOT use serialNumber values longer than 20 octets.
Note: Non-conforming CAs may issue certificates with serial numbers that are negative or zero. Certificate users SHOULD be prepared to gracefully handle such certificates”

In addition, section 7.1 of the BR indicates the following:

“Effective September 30, 2016, CAs SHALL generate non‐sequential Certificate serial numbers greater than zero (0) containing at least 64 bits of output from a CSPRNG.”

PSC PROCERT works with the Microsoft cryptography service, which has a CSPRNG inside the CryptoAPI library suite, which includes a CryptGenRandom function, which is a cryptographically secure pseudo-random number generator, this function was found by default in the generation of the short serial numbers, therefore we proceeded to modify the registry of the CA and activate the option of high serial, which comes by default deactivated (0), we proceeded to activate this registry, so that serials are generated under the parameters of the standard.

In the following link you can see an example of a certificate with the appropriate serial number

https://crt.sh/?id=204446748

After this action was taken, we proceeded to recognize the certificates with these problems and were notified to our clients that they should be revoked and reissued, the certificates denounced in the bug are revoked.

PSC PROCERT is not the only one to present this case, QuoVadis and SwissSign, presented the same situation and the remediation was accepted.

https://bugzilla.mozilla.org/show_bug.cgi?id=1391063

https://bugzilla.mozilla.org/show_bug.cgi?id=1391066


Note that the answers offered by QuoVadis and SwissSign were simple and not detailed; such as those offered by PROCERT, the response and follow-up on compliance was further expanded. We do not understand then why for other cases apply and for PROCERT not ?.

2.- Issues with SSL Certificates
Issue D: URI in CN and dnsName SAN (December 2016)
Issue G: Internal IP Address in SAN (March 2015 - March 2017)
Issue I: CN Not Also In SAN (March 2016 - June 2017)
Issue K: Internal DNS Names in Certificates (May - June 2017)
Issue L: helloburgershack.com (June - July 2017)

2.1. Issue R: Incorrect Encoding of or Inappropriate Use of TeletexString (December 2015 - August 2017)
Taking into account what was stated in the bug, the BR was reviewed and it indicates the following in section 7.1.4.2.2

“j. Other Subject Attributes
All other optional attributes, when present within the subject field, MUST contain information that has been verified by the CA. Optional attributes MUST NOT contain metadata such as ‘.’, ‘‐‘, and ‘ ‘ (i.e. space) characters, and/or any other indication that the value is absent, incomplete, or not applicable.e.”

In addition, section 7.4.2.1 states the following

“7.1.4.2.1. Subject Alternative Name Extension
Certificate Field: extensions:subjectAltName
Required/Optional: Required
Contents: This extension MUST contain at least one entry. Each entry MUST be either a dNSName containing the Fully‐Qualified Domain Name or an iPAddress containing the IP address of a server.
The CA MUST confirm that the Applicant controls the Fully‐Qualified Domain Name or IP address or has been granted the right to use it by the Domain Name Registrant or IP address assignee, as appropriate.”

In order to remedy this situation, the affected customers were notified that the certificates had to be revoked and issued again, modifying the data with problems, then notified and with the window of time for customers to take these changes into account. level of their systems, the certificates were revoked and issued correctly.

To avoid these errors in the future, PSC PROCERT proceeded to modify the CPS and PC, adding a section to inform our customers that any request for SSL certificate must comply with the standard of the CA Browser Forum, also within our system a filter was applied to avoid accepting the following parameters

1. Characters "-", ",", ".", ":", "/", And " " (Issue D)
2. Private IP addresses (Issue G)
3. Domains ending in * .local (Issue K)
4. Character accentuation (eg or) (Issue R)
5. Validator that both the CN and the SAN are the same values (Issue I)

In addition to establishing internal review and validation mechanisms with the personnel that analyze the CSR and support our clients. A tool will also be incorporated to automate the analysis of CSRs. Such software is currently being tested in our quality environment to enter production.

2.2. Issue N: Other Names in Certificate SAN (2011 - August 2017
Referring to Issue N, the BR was reviewed and it was found that section 7.1.4.2.2 section i indicates the following
“i. Other Subject Attributes
All other optional attributes, when present within the subject field, MUST contain information that has been verified by the CA.”

This indicates that another field can be included as long as it is information verified by the CA, in this case PSC PROCERT verifies the number of RIF that is the tax identification number of company in Venezuela of each company, which by definition is a registry destined to the legal control of taxes and in which natural or juridical persons, communities and entities or groups without legal personality, susceptible by reason of the goods or activities, of being subject or responsible for the Income Tax, the tax retention agents, and residents abroad without permanent establishment or fixed base, provided that the cause of the enrichment is or occurs in Venezuela. In conclusion, it is a requirement of Law in Venezuela and the company that omits to place it is sanctioned. Even PSC PROCERT can be sanctioned by the government in case of failure to include such information in the certificates.

Each company that signs a contract with PSC PROCERT must present a copy of the RIF and the same is proceeded to validate against the national tax office, which in the case of Venezuela is SENIAT; The SUSCERTE that is the governing entity that regulates in Venezuela requires that in the structure of the certificate this information is placed in a field of the certificate identified with an OID for this purpose, which is 2.16.862.2.

Therefore, it can be concluded that the standard is fulfilled and there is no default as such in Issue N.

There are similar cases with accepted remediation
CertSign
https://bugzilla.mozilla.org/show_bug.cgi?id=1390979

GoDaddy
https://bugzilla.mozilla.org/show_bug.cgi?id=1391429

Entrust
https://bugzilla.mozilla.org/show_bug.cgi?id=1390996

The responses provided by PSC PROCERT are in line with those provided for similar cases by the CA’s above.

3. Issue T: Inappropriate Key Usage Value of "Key Agreement" (October 2016 - August 2017)
Certificates with this Issue were revoked and notified to customers. In order to prevent this situation from being repeated, we proceeded to review the template with which the certificates were issued and the key Agreement was eliminated among the uses, in addition it was enabled so that this option can NOT be used later

4. Issue V: Failure to Respond Quickly To Problem Reports (August 2017)
At this point and about the rapid response, we point out that at all times we have demonstrated since we were notified, willingness to solve this problem. Certificates that did not generate impact were automatically revoked.

Please find attached our CPS and the evidence of SSL revocation.

Please consider this evidence in order to reopen the PSC PROCERT case.

alejand...@gmail.com

unread,
Oct 2, 2017, 11:18:46 AM10/2/17
to mozilla-dev-s...@lists.mozilla.org

Kathleen Wilson

unread,
Oct 2, 2017, 1:43:02 PM10/2/17
to mozilla-dev-s...@lists.mozilla.org
On Friday, September 29, 2017 at 2:52:49 PM UTC-7, Eric Mill wrote:
> That dynamic is natural, but accepting that this dynamic exists is
> different than giving into it in some absolute way. When offering second
> chances, requiring that the person/org fulfill certain conditions that
> speak directly to their ability to have learned and adapted from the thing
> they failed at the first time is an approach that accepts this dynamic,
> without shutting the door on people or organizations that have grown as a
> result of the experience.
>
> I think it would arguably lead to worse behavior, and less disclosure of
> incidents and mistakes, if Mozilla adopted a posture where second chances
> are rarely given. Not saying that's what's being said here, but I think
> it's worth emphasizing that the first principle here should be to optimize
> for incentivizing the behavior you want out of the CA community that
> protects users and increases information sharing.
>
> -- Eric


I agree with Eric on this.

The last sentence in our CA Communications and Mozilla Security Blog Posts regarding our CA Program is frequently:
"... we believe that the best approach to safeguard that security is to work with CAs as partners, to foster open and frank communication, and to be diligent in looking for ways to improve."

Below are my personal feelings...

In the case of WoSign and StartCom, I felt such a level of deception that it will be extremely difficult for either CA to ever gain my trust again. Rightly or wrongly so, I have not recognized that level of deception from other CAs in Mozilla's program. The deception happened before Inigo went to StartCom, and I appreciate all of Inigo's efforts, but due to the position he is now in, he will have to do an outstanding job, be test-driven, and demonstrate a truly clean CA hierarchy in order to regain my trust in StartCom. Unfortunately, I just don't feel that I've seen that so far.

In regards to PROCERT, I do not believe that they have intentionally deceived me, but that their representatives responded to previous CA Communications and the Bugzilla Bug without getting their technical people involved. That is very bad! and I am very disappointed! But perhaps there are actions that they can take to demonstrate their commitment to not repeating those mistakes, to putting code into place to prevent non-BR-compliant cert issuance, and to show that they do have the level of technical knowledge in their organization that is needed to operate a good CA.

Thanks,
Kathleen





Ryan Sleevi

unread,
Oct 2, 2017, 6:15:00 PM10/2/17
to Kathleen Wilson, mozilla-dev-security-policy
On Mon, Oct 2, 2017 at 10:42 AM, Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Friday, September 29, 2017 at 2:52:49 PM UTC-7, Eric Mill wrote:
> > That dynamic is natural, but accepting that this dynamic exists is
> > different than giving into it in some absolute way. When offering second
> > chances, requiring that the person/org fulfill certain conditions that
> > speak directly to their ability to have learned and adapted from the
> thing
> > they failed at the first time is an approach that accepts this dynamic,
> > without shutting the door on people or organizations that have grown as a
> > result of the experience.
> >
> > I think it would arguably lead to worse behavior, and less disclosure of
> > incidents and mistakes, if Mozilla adopted a posture where second chances
> > are rarely given. Not saying that's what's being said here, but I think
> > it's worth emphasizing that the first principle here should be to
> optimize
> > for incentivizing the behavior you want out of the CA community that
> > protects users and increases information sharing.
> >
> > -- Eric
>
>
> I agree with Eric on this.
>
> The last sentence in our CA Communications and Mozilla Security Blog Posts
> regarding our CA Program is frequently:
> "... we believe that the best approach to safeguard that security is to
> work with CAs as partners, to foster open and frank communication, and to
> be diligent in looking for ways to improve."
>
> Below are my personal feelings...
>
> In the case of WoSign and StartCom, I felt such a level of deception that
> it will be extremely difficult for either CA to ever gain my trust again.
> Rightly or wrongly so, I have not recognized that level of deception from
> other CAs in Mozilla's program. The deception happened before Inigo went to
> StartCom, and I appreciate all of Inigo's efforts, but due to the position
> he is now in, he will have to do an outstanding job, be test-driven, and
> demonstrate a truly clean CA hierarchy in order to regain my trust in
> StartCom. Unfortunately, I just don't feel that I've seen that so far.
>
> In regards to PROCERT, I do not believe that they have intentionally
> deceived me, but that their representatives responded to previous CA
> Communications and the Bugzilla Bug without getting their technical people
> involved. That is very bad! and I am very disappointed! But perhaps there
> are actions that they can take to demonstrate their commitment to not
> repeating those mistakes, to putting code into place to prevent
> non-BR-compliant cert issuance, and to show that they do have the level of
> technical knowledge in their organization that is needed to operate a good
> CA.
>

These are also my personal feelings, and not speaking on behalf of Mozilla
or Google.

I think these are excellent ways of framing the discussion, given the
current state of the ecosystem and the incentives. I think attempting to
blackball a CA organization is largely ineffective, for reasons others have
highlighted - such as the lack of transparency around beneficial or
controlling interests.

With respect to PROCERT, I think it's important to note that there are
several problematic practices:
- The displayed technical competence was grossly insufficient for the level
of trust afforded.
- The technical responses (continue) to demonstrate a degree of
interpretative leeway that is not supported by the text.
- The sum totality of the number and nature of the incidents raise serious
concerns about the overall operation.

Unlike other incidents, there's no one "Fix this thing" incident - it's a
systemic failure of both implementation and oversight that caused the loss
of a trust, a dozen of incidents both big and small, and the response to
those incidents, that caused a loss of trust.

I think it's reasonable to suggest that developing technical competence is
not something that will happen overnight. And systemically addressing the
organizational and operational failures will require a careful degree of
analysis of the past issues and preventive steps (of which revocation is,
again, completely insufficient).

We've also seen from past incidents and discussions that one year is very
likely too short a time to allow such remediations. If anything, it
incentivizes hasty changes, rather than methodical analysis. It's very
likely that, for an organization like PROCERT, it would be far more
appropriate to suggest something like three or more years - to allow them
the opportunity to invest and improve.

Similarly, given the nature of these incidents, it seems reasonable to
suggest that new keys must be used, without any relation to the old keys or
infrastructure. This hopefully goes without saying.

In terms of oversight and audits, it seems beneficial to have a
community-agreed upon auditor, a Mozilla-delegated auditor, or even an open
RFP, so that auditors themselves can compete on the thoroughness and
oversight that they can provide. Similarly, it seems reasonable to expect a
more detailed audit report, in which management attests to their system
design, and the controls they have in place to ensure compliance, as well
as the auditor's evaluation of those controls (including all controls the
auditor evaluated, not just those that were successfully validated). This
assertion by management - and the auditors independent assessment of the
factual veracity of that assertion - can help provide a greater level of
assurance that PROCERT has successfully understood and integrated the BRs.

These are just some ideas on concrete steps to restore trust when there is
a systemic pattern of failures and misunderstandings.

Kathleen Wilson

unread,
Oct 3, 2017, 1:39:07 PM10/3/17
to mozilla-dev-s...@lists.mozilla.org
Here's a draft of the Bugzilla Bug that I plan to file to list the action items for PROCERT to complete before they may re-apply for inclusion in Mozilla's Root Store. I will appreciate feedback on this.

== DRAFT ==
Subject: PROCERT: Action Items

As per Bug #1403549 the PSCProcert certificate will be removed from Mozilla’s Root Store due to a long list of problems and they way that PROCERT responded to those problems (and to previous CA Communications). For details about the problems, see Bug #1391058 and https://wiki.mozilla.org/CA:PROCERT_Issues

The purpose of this bug is to record the action items that PROCERT must complete before their certificate may be included as a trust anchor in Mozilla’s Root Store again.

PROCERT may apply for inclusion of a new certificate[1] following Mozilla's normal root inclusion/change process[2], after they have completed all of the following action items.

1. Provide a list of changes that the CA plans to implement to ensure that there are no future violations of Mozilla's CA Certificate Policy and the CA/Browser Forum's Baseline Requirements.

2. Implement the changes, and update their CP/CPS to fully document their improved processes.

3. Provide a reasonably detailed[4] public-facing attestation from a licensed auditor[3] acceptable to Mozilla that the changes have been made. This audit may be part of an annual audit.

4. Provide auditor[3] attestation that a full performance audit has been performed confirming compliance with the CA/Browser Forum's Baseline Requirements. This audit may be part of an annual audit.


Notes:
[1] The new certificate (trust anchor) may be cross-signed by the removed certificate. However, the removed certificate may *not* be cross-signed by the new certificate, because that would bring the concerns about the removed certificate into the scope of the new trust anchor.
[2] Mozilla's root inclusion/change process includes checking that certificates in the CA hierarchy comply with the CA/Browser Forum's Baseline Requirements.
[3] The auditor must be an external company, and approved by Mozilla.
[4] “detailed” audit report means that management attests to their system design and the controls they have in place to ensure compliance, and the auditor evaluates and attests to those controls. This assertion by management - and the auditor's independent assessment of the factual veracity of that assertion - will help provide a greater level of assurance that PROCERT has successfully understood and integrated the BRs.
==

Thanks,
Kathleen

Ryan Sleevi

unread,
Oct 3, 2017, 4:07:27 PM10/3/17
to Kathleen Wilson, mozilla-dev-security-policy
Hi Kathleen,

With respect to providing a list - is there any requirement to ensure
Mozilla accepts that as a reasonable remediation?

For example, would "We plan to not do the same in the future" be an
acceptable remediation plan? As currently worded, it would seem to meet the
letter of this requirement.

This would be useful to ensure before [2]
> auditor evaluates and attests to those controls. This assertion by
> management - and the auditor's independent assessment of the factual
> veracity of that assertion - will help provide a greater level of assurance
> that PROCERT has successfully understood and integrated the BRs.
> ==
>
> Thanks,
> Kathleen

Kathleen Wilson

unread,
Oct 4, 2017, 5:57:23 PM10/4/17
to mozilla-dev-s...@lists.mozilla.org
Bug Filed regarding PROCERT Action Items:
https://bugzilla.mozilla.org/show_bug.cgi?id=1405862

Thanks,
Kathleen

Gervase Markham

unread,
Oct 4, 2017, 11:05:47 PM10/4/17
to mozilla-dev-s...@lists.mozilla.org
On 05/10/17 05:57, Kathleen Wilson wrote:
> Bug Filed regarding PROCERT Action Items:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1405862

Hi Kathleen,

I know you have already filed the bug, but I think that perhaps the list
of action items might need to be a bit more detailed and/or rigorous
than it is at the moment.

For example, I think there is wisdom in what Ryan says about setting an
amount of time before a company can re-apply. In the case of StartCom we
did not set such a time, because I had thought they might do what I
recommended, which was to switch back from the new WoSign infra that we
didn't trust to the original StartCom infra, which we did. However, they
instead chose to implement new infra from scratch and rushed it, with
the result being the use of PHP, the use of coders without sufficient
training in security, and some terrible code written under extreme time
pressure driven by commercial considerations.

We did give WoSign a time limit of 1 year, although it seems that they
(now called WoTrus) have not yet applied for re-inclusion.

So I think it would be appropriate for us to set a minimum period before
PROCERT can re-apply, and that it be longer than 1 year.

In addition, we do need to address the question of how we can ascertain
that the organization has acquired the technical competence and
management rigour which seems to be lacking. I know you have placed some
audit requirements in there, but I do think we need to discuss whether
those will provide a sufficient guarantee and, if not, if there's
anything that could.

Gerv

Matt Palmer

unread,
Oct 5, 2017, 3:09:36 AM10/5/17
to dev-secur...@lists.mozilla.org
On Thu, Oct 05, 2017 at 11:05:07AM +0800, Gervase Markham via dev-security-policy wrote:
> In addition, we do need to address the question of how we can ascertain
> that the organization has acquired the technical competence and
> management rigour which seems to be lacking. I know you have placed some
> audit requirements in there, but I do think we need to discuss whether
> those will provide a sufficient guarantee and, if not, if there's
> anything that could.

One thing that might provide at least a certain amount of motivation, if not
guarantee, would be to apply an additional delay in re-application
processing if review of the initial re-application is found to be deficient.
I know a similar thing has been suggested in the past for applications in
general, to avoid the situation where new applications are essentially being
taught webPKI by the participants in the Mozilla application process,
although I don't recall off the top of my head what the concerns were in
general.

However, for a re-application of this sort, a "get it right or face delay"
would send a strong message that the applicant really needs to spend the
time to get their ducks in a row before coming back. Otherwise, they could
very easily spend the entire next twelve months sitting on their hands, then
come back with the same infrastructure and play whack-a-mole with whatever
gets identified until everyone else gets tired and gives up finding more
holes. If they know they'll face another three, six, or even twelve month
delay if anything untoward is found, it might encourage them to look a
little harder. I'd like to include audit findings in that, although I'm
aware that audits, being controlled by the company under audit, can be
suppressed if they're not to the company's liking.

There is also the possibility of requiring an audit (not necessarily a
BR/WTCA audit; it could be any form of review based on identified
deficiencies in the organisation) by a Mozilla-selected auditor, *reporting
to Mozilla* (not the applicant), and paid for by the applicant. This has a
degree of precedent in the StartCom case, where Mozilla required a code
audit (above and beyond the usual BR/WTCA ones). There's also precedent in
financial auditing, I believe, although I can't find the reference I recall
reading (and thinking, "hmm, could that work for CAs?"). Whilst it might be
considered an excessive burden in general (even on a "random selection"
basis), for an applicant who has been previously distrusted, I don't think
it's out of the realm of possibility to say, "sorry, we're going to need a
bit more from you".

Sure, we can hypothesise about organisations doing shady reorgs to try and
look like they're "new". However, the same problems could apply now --
Procert (or any other distrusted org) could try and get around whatever
sanctions have been imposed (such as a re-application delay) by re-forming
as a seemingly-unrelated entity. Based on the impressive investigations
that have been undertaken in the past, I'm fairly confident that most forms
of shenanigan could be detected. If Mozilla has "reasonable confidence"
that an applicant shares key personnel, or a controlling entity, with a
previously distrusted organisation, I don't think it's unreasonable to stick
then with the cost of an additional review of their previously-unacceptable
infrastructure and processes.

- Matt

Inigo Barreira

unread,
Oct 5, 2017, 3:26:34 AM10/5/17
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
>
> For example, I think there is wisdom in what Ryan says about setting an
> amount of time before a company can re-apply. In the case of StartCom we
> did not set such a time, because I had thought they might do what I
> recommended, which was to switch back from the new WoSign infra that we
> didn't trust to the original StartCom infra, which we did. However, they
> instead chose to implement new infra from scratch and rushed it, with the
> result being the use of PHP, the use of coders without sufficient training
in
> security, and some terrible code written under extreme time pressure
driven
> by commercial considerations.
>

All that "terrible" code written was before we went live and before we
applied for re-inclusion.
That´s not the code we use to issue certificates, in fact, the hierarchy was
not built at that time so with this comment people may think that we´re
using a bad code which is not true. What it´s true is that we wanted to
re-apply the sooner because yes, we had some commercial issues, and we did
have a timeline, which was 6 months.
The reason for not using the old startcom code was due to some past issues
arised during the time but the new code is much more secure than the old one
and with more functionalities as have been explained.
The misissuances we´ve made were before the re-application and none due to a
bad code.
I know this reply is not related to the email thread but wouldn´t like to
leave the feeling that the code we are using is bad, or not secure, etc.

Gervase Markham

unread,
Oct 5, 2017, 5:49:01 AM10/5/17
to mozilla-dev-s...@lists.mozilla.org
On 05/10/17 15:32, Inigo Barreira wrote:
> I know this reply is not related to the email thread but wouldn´t like to
> leave the feeling that the code we are using is bad, or not secure, etc.

Perhaps now might be a good time to publish the security audits that
were done on the code, then?

Gerv

Inigo Barreira

unread,
Oct 5, 2017, 7:55:02 AM10/5/17
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
Has this been asked ever? Has any other CA published it? It´s just to know. And, is there a "default" scope for this kind of security audits?

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

> -----Original Message-----
> From: dev-security-policy [mailto:dev-security-policy-
> bounces+inigo=startco...@lists.mozilla.org] On Behalf Of Gervase
> Markham via dev-security-policy
> Sent: jueves, 5 de octubre de 2017 11:48
> To: mozilla-dev-s...@lists.mozilla.org
> Subject: Re: PROCERT issues
>
> On 05/10/17 15:32, Inigo Barreira wrote:
> > I know this reply is not related to the email thread but wouldn´t like
> > to leave the feeling that the code we are using is bad, or not secure, etc.
>
> Perhaps now might be a good time to publish the security audits that were
> done on the code, then?
>
> Gerv

westm...@gmail.com

unread,
Oct 5, 2017, 8:36:27 AM10/5/17
to mozilla-dev-s...@lists.mozilla.org
This whole discussion is very interesting, but as an ordinary user of your root storage I would like to say that I deleted all root certificates of WoSign, StartCom/Camerfirma A.S, also root certificates of Certinomis and Certum CA from all my of their root stores, as they are cross for StartCom and in the next 4 years I definitely will not keep them.

Enjoy,
Andrew.

Gervase Markham

unread,
Oct 5, 2017, 8:36:45 AM10/5/17
to mozilla-dev-s...@lists.mozilla.org
On 05/10/17 20:00, Inigo Barreira wrote:
> Has this been asked ever? Has any other CA published it? It´s just to know. And, is there a "default" scope for this kind of security audits?

Well, you indicated your willingness to publish them in an email to me,
if I remember correctly. And it would allow others to assess your claim
that the code is not bad, and is secure.

Gerv

okaphone.e...@gmail.com

unread,
Oct 5, 2017, 11:19:54 AM10/5/17
to mozilla-dev-s...@lists.mozilla.org
On Thursday, 5 October 2017 13:55:02 UTC+2, Inigo Barreira wrote:
> Has this been asked ever? Has any other CA published it? It´s just to know. And, is there a "default" scope for this kind of security audits?

Grin. ;-)

Does it matter? Or perhaps more important, do you want to recover from loosing your trust? Because if so, then it might be a good idea to consider where this thing called trust is coming from. And perhaps also the increasing significance of the word transparency in web trust.

CU Hans

Kathleen Wilson

unread,
Oct 9, 2017, 2:57:31 PM10/9/17
to mozilla-dev-s...@lists.mozilla.org
Here's what is currently in the bug...
https://bugzilla.mozilla.org/show_bug.cgi?id=1405862
~~
As per Bug #1403549 the PSCProcert certificate will be removed from Mozilla’s Root Store due to a long list of problems and the way that PROCERT responded to those problems (and to previous CA Communications). For details about the problems, see Bug #1391058 and https://wiki.mozilla.org/CA:PROCERT_Issues

The purpose of this bug is to record the action items that PROCERT must complete before their certificate may be included as a trust anchor in Mozilla’s Root Store again.

PROCERT may apply for inclusion of a new certificate[1] following Mozilla's normal root inclusion/change process[2], after they have completed all of the following action items.

1. Provide a list of changes that the CA plans to implement to ensure that there are no future violations of Mozilla's CA Certificate Policy and the CA/Browser Forum's Baseline Requirements. The list must be provided in this bug, and accepted by Mozilla as reasonable remediation.

2. Implement the changes, and update their CP/CPS to fully document their improved processes.

3. Provide a reasonably detailed[4] public-facing attestation from a licensed auditor[3] acceptable to Mozilla that the changes have been made. This audit may be part of an annual audit.

4. Provide auditor[3] attestation that a full performance audit has been performed confirming compliance with the CA/Browser Forum's Baseline Requirements. This audit may be part of an annual audit.


Notes:
[1] The new certificate (trust anchor) may be cross-signed by the removed certificate. However, the removed certificate may *not* be cross-signed by the new certificate, because that would bring the concerns about the removed certificate into the scope of the new trust anchor.
[2] Mozilla's root inclusion/change process includes checking that certificates in the CA hierarchy comply with the CA/Browser Forum's Baseline Requirements.
[3] The auditor must be an external company, and approved by Mozilla.
[4] “detailed” audit report means that management attests to their system design and the controls they have in place to ensure compliance, and the auditor evaluates and attests to those controls. This assertion by management - and the auditor's independent assessment of the factual veracity of that assertion - will help provide a greater level of assurance that PROCERT has successfully understood and integrated the BRs.
~~

I could add a comment to the bug saying:
~~
To be clear…

- This PSCProcert certificate will be removed, and the CA has to re-apply for inclusion of the new certificate through Mozilla’s application process as described here:
https://wiki.mozilla.org/CA/Application_Process#Process_Overview
However, the CA must not re-apply until all of the listed actions have been completed, related documents provided in this bug, and resulting documents/audits approved by Mozilla in this bug.

- Action #3 must not start until Action #1 and Action #2 are done, and the resulting documents approved by Mozilla in this bug.

- As with all root inclusion applications, the new CA hierarchy, processes, issued certificates, CP/CPS, BR Self Assessment, and audits must fully comply with Mozilla’s Root Store Policy and the CA/Browser Forum’s Baseline requirements. Otherwise, there will be further delay.
~~

Does that make it clear enough that this is not something to be rushed?
And that the items each need to be approved by Mozilla in the bug?

I'm not fond of the idea of stating time delays up front, when our application process is by-design very slow and cumbersome already. (Just ask the CAs who have been waiting for over a year for the discussion of their root inclusion/update requests to start.)

I think the "get it right or face delay" concept is already a standard part of our application process.

Action #3 already requires an audit statement about the changes.

I'm not comfortable with the idea of having an auditor report to Mozilla and not the applicant, because I would suspect that might require some contractual agreements and/or NDAs, and I do not sign contractual agreements or NDAs with CAs.

Thanks,
Kathleen



Message has been deleted

James Burton

unread,
Oct 9, 2017, 3:33:25 PM10/9/17
to mozilla-dev-s...@lists.mozilla.org
Hi Kathleen,

I think there should be a comment relating to whether PROCERT is allowed/disallowed to cross-sign their new root CA with another CA.

James
0 new messages