Recent Entrust Compliance Incidents

[Ben Wilson]

Dear Mozilla Community,

Over the past couple of months, a substantial number of compliance incidents have arisen in relation to Entrust. We have summarized these recent incidents in a dedicated wiki page: https://wiki.mozilla.org/CA/Entrust_Issues. In brief, these incidents arose out of certificate mis-issuance due to a misunderstanding of the EV Guidelines, followed by numerous mistakes in incident handling (including a deliberate decision to continue mis-issuance), which have been compounded by a failure to remediate the issues in a timely fashion in line with well-established norms and root store requirements.

Our preliminary assessment of these incidents is that while they were relatively minor initially, the poor incident response has substantially aggravated them and the progress towards full remediation remains unacceptably slow. This is particularly disappointing in light of previous incidents in 2020 (#1651481 and #1648472), which arose out of similar misunderstandings of the requirements, similar poor decision-making in the initial response, and lengthy remediation periods that fell well below expectations. Entrust gave commitments in those bugs to address the root problems through process improvements, and it is concerning to see so little improvement 4 years later.

In light of these recent incidents, we are requesting that Entrust produce a detailed report of them. This report should cover in detail: 

• The factors and root causes that lead to the initial incidents, highlighting commonalities among the incidents and any systemic failures;

• Entrust’s initial incident handling and decision-making in response to these incidents, including any internal policies or protocols used by Entrust to guide their response and an evaluation of whether their decisions and overall response complied with Entrust’s policies, their practice statement, and the requirements of the Mozilla Root Program;

• A detailed timeline of the remediation process and an apportionment of delays to root causes; and 

• An evaluation of how these recent issues compare to the historical issues referenced above and Entrust’s compliance with its previously stated commitments. 

Finally, Entrust’s report should include a detailed proposal on how it plans to address the root causes of these issues. In light of previous guarantees given by Entrust in 2020 to ensure speedy remediation in future incidents, this proposal should include:

• Clear and concrete steps that Entrust proposes to take to address the root causes of these incidents and delayed remediation;

• Measurable and objective criteria for Mozilla and the community to evaluate Entrust’s progress in deploying these solutions; and

• A timeline for which Entrust will commit to meeting these criteria.

We strongly recommend that Entrust go beyond their existing commitment to offer systematic, automated solutions for effective remediation, like ACME ARI and that it also include clear and measurable targets for the adoption of these tools by new and existing subscribers. 

This report should be submitted to Mozilla dev-security-policy mailing list for evaluation by the community and Mozilla, who will weigh whether Entrust’s report presents a credible and effective path towards re-establishing trust in Entrust’s operation. Submission should be no later than June 7, 2024.

We thank community members for their engagement on these issues and look forward to their feedback on Entrust’s report and proposed commitments.

 Thanks,

Ben Wilson

Mozilla Root Program

[Watson Ladd]

Could we add a section for geographical incidents? This is slightly
outside your time window, but I think reading the series here has some
uncanny echos in the ones in your window.

https://bugzilla.mozilla.org/show_bug.cgi?id=1658792
https://bugzilla.mozilla.org/show_bug.cgi?id=1658794
https://bugzilla.mozilla.org/show_bug.cgi?id=1802916
https://bugzilla.mozilla.org/show_bug.cgi?id=1804753
https://bugzilla.mozilla.org/show_bug.cgi?id=1867130

On Tue, May 7, 2024 at 7:59 AM 'Ben Wilson' via
dev-secur...@mozilla.org <dev-secur...@mozilla.org>
wrote:

> --
> You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
> To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaYURqFzRqVmJdc7fBXE1mbGs25HpSkp5wZ0Xm%2BRG0YHCA%40mail.gmail.com.



--
Astra mortemque praestare gradatim

[Ben Wilson]

Here are draft summaries of the additional historic incidents. I'll be adding these to the Entrust Issues page: https://wiki.mozilla.org/CA/Entrust_Issues

Invalid data in State/Province Field -

https://bugzilla.mozilla.org/show_bug.cgi?id=1658792

It was initially discovered that Entrust had issued 395 OV SSL certificates to a large international organization with “NA” for the state/province information. Entrust worked on a drop-down list to prevent the error. Certificate revocation would not occur within established timeframes, so Bug #1658794 for delayed revocation was opened. 

Late Revocation for Invalid State/Province Issue -
https://bugzilla.mozilla.org/show_bug.cgi?id=1658794

This is the delayed revocation bug related to Bug #1658792, above. Entrust said that when educating large institutions about rapid revocation, factors include who owns a certificate, where it is deployed, and the type of system or application that requires the certificate.  It also said that it was advocating automation with such institutions to help speed up certificate replacement and to minimize human error.

EV TLS Certificate incorrect jurisdiction -

https://bugzilla.mozilla.org/show_bug.cgi?id=1802916

Entrust mis-issued 322 EV certificates with the wrong state and locality jurisdiction fields due to complex data entry processes. Entrust implemented a different automated dropdown system for jurisdiction selection. Certificate revocation would not occur within established timeframes, so Bug #1804753 for delayed revocation was opened. 

Delayed Revocation for EV TLS Certificate incorrect jurisdiction -

https://bugzilla.mozilla.org/show_bug.cgi?id=1804753

This is the delayed revocation bug related to Bug #1802916, above. Entrust listed 8 Subscribers who were pushing back on immediate certificate revocation and the reasons given (e.g. extensions granted due to end-of-year freezes). Entrust committed to “continue to develop and extend methods for automatic certificate renewal.”

Jurisdiction Locality Wrong in EV Certificate -

https://bugzilla.mozilla.org/show_bug.cgi?id=1867130

Two EV TLS Certificates were mis-issued due to human error in the Jurisdiction Locality field. (The incident revealed 340 additional accounts needing similar updates.) Entrust said it would enhance its linting processes to include possibly using an external service to validate locality data against verified country data.

SHA-256 hash algorithm used with ECC P-384 key -

https://bugzilla.mozilla.org/show_bug.cgi?id=1648472

A Mozilla policy was adopted to require hashing with SHA-384 for an ECC P-384 key. Existing CAs using SHA-256 were not re-configured when Mozilla adopted this policy.  This incident revealed a serious gap in taking new requirements and implementing them. Ryan Sleevi noted that linting was just a safety net and not a systemic solution. Entrust was also criticized for the lack of detail in its incident report and its decision to not revoke the certificates.

Entrust committed to improving its monitoring and implementation of policy changes to prevent similar incidents. Ryan set forth a number of proactive systemic corrections that Entrust needed to take, rather than taking a reactive stance on matters of non-compliance.

Entrust committed to rigorous review of certificate profiles, browser policy revisions, and industry developments. As a final comment, Ryan said, “My big concern is, going forward, we see incident reports from Entrust take a more systemic, holistic response, like Comment #16, to try and cover the scenarios, and to provide sufficient detail about the situation and its failures to understand how those relate. The goal isn't to make CAs wear proverbial sackcloth, it's to try and make sure we're understanding how things go wrong, so that we can effectively collaborate on identifying solutions to avoid that going forward.”

Late Revocation due to SHA-256 hash algorithm -

https://bugzilla.mozilla.org/show_bug.cgi?id=1651481

This bug is related to Bug #1648472.  Entrust issued TLS certificates using ECC P-384 keys hashed with SHA-256, contrary to Mozilla policy requiring SHA-384 for hashing. Entrust’s initial decision was to allow certificates to expire naturally without revocation, but this was revised with a decision to revoke all affected certificates. Entrust committed to: filing incident report within one business day for future incidents, filing late revocation incident reports within the required 24 hours or 5 days, as applicable, and advising Subscribers about revocation within 24 hours or 5 days, or provide an explanation if they are unable to meet such timeframes. Entrust was told it needed to align its revocation procedures more closely with the Baseline Requirements and Mozilla’s policy, especially in providing a detailed rationale for any delays in revocation on a per-subscriber basis and ensuring timely revocation in line with the Baseline Requirements.

 

[George]

Although it was not mentioned in the original bug, it may be worth adding that the certificates in bug 1867130 were also not revoked within 5 days of discovery. Entrust might've based the start of the 5 day deadline at the time the "Director of compliance confirmed investigation conclusions to support team" at 2023-11-21 15:00 UTC with all certificates being revoked by 2023-11-26 14:50 UTC, but I don't think that's correct if that was the case.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtabu3a3FRyG4C%3DDdXDSAAMbdPVKApACYnE%2Be2tFCsdULSQ%40mail.gmail.com.

[Ben Wilson]

Added " Although not expressed in the bug, it appears that certificate revocation was delayed as well."

[Chris Bailey]

To Ben Wilson and the Mozilla Community:

 

I want to acknowledge your letter and the input from you and the community. We agree that we have go-forward opportunities to improve.

 

To that end, I want to confirm our intent to provide a full written response to you and the community prior to June 7. Until then, please contact me directly with additional questions or feedback.

 

Sincerely,

Chris Bailey

VP-Digital Certificates

Entrust

Any email and files/attachments transmitted with it are intended solely for the use of the individual or entity to whom they are addressed. If this message has been sent to you in error, you must not copy, distribute or disclose of the information it contains. Please notify Entrust immediately and delete the message from your system.

[Wayne]

I can't speak for everyone but in an issue of public trust asking for private feedback and concerns is not helping matters.

One of the prevalent topics that have came up in these issues is shorter certificate lifespans, certificate automation and how Entrust are working very hard with their customers. I'm very curious if Entrust can quantify this in any way?

Taking a step back and just looking at their public statements regarding lifespans, automation and ACME should give us an idea of their internal viewpoint and how this topic is presented to customers. Outside of the first issue I won't be quoting bugzilla, it's there solely to provide context as I can't see an earlier point that automation was promised.

Let's dive in, sources all provided if there are any questions about the rough transcript or context.

1: 2023-03-27: Entrust: Delayed Revocation for EV TLS Certificate incorrect jurisdiction
https://bugzilla.mozilla.org/show_bug.cgi?id=1804753#c8
---
Late revocations are base primarily by Subscribers which have not implemented automation. We have increased our efforts to extend implementation of ACME. We are also considering implementing the ACME Renewal Information (ARI) Extension, which allow the certificate to be automatically renewed before the revocation date.

In the shorter term, we will provide Subscribers with automated reminders on the revocation date for miss-issued certificates. We will plan to allow short extensions, based on the severity of the miss-issuance.
---



2: 2023-04-21: Google’s 90 day proposal for TLS certificates
https://www.entrust.com/blog/2023/04/googles-90-day-proposal-for-tls-certificates/
---
Even if CAs and other browsers don’t share Google’s objectives, there is a chance that Google could unilaterally make this change in its root program and force the entire industry in this direction in a time of their choosing. We hope that browsers will not make this decision unilaterally, but instead allow the decision to be made with broad industry and website owner consensus.

Another issue is that Google has presented no public research or factual data showing that such a change to the ecosystem is necessary or useful in many use cases. We believe there will be much discussion before a 90-day ballot will pass at the CABF as several CAs have indicated that a requirement for 90-day certificates might have far-reaching implications. There have also been several EU governmental bodies concerned over the market and competitive implications of Google’s proposal and the impact on eIDAS Qualified Website Authentication Certificate objectives, which are now being reasserted in the EU’s update of its eIDAS legislation.

Entrust does not believe that a maximum 90-day limit for TLS certificate lifetimes is the only method to drive automation and the deployment of ACME. Additionally, Entrust doesn’t believe that ACME is the only method for automation, or that it would be accepted by some of the most complex subscriber secure server deployments. Rather, we believe subscribers should be encouraged to deploy automation, but do not need to be discouraged by the cost and complexity of certificates with 90-day maximum validity.

While Entrust is not currently in favor of a mandatory 90-day certificate limit, we have no objection to 90-day certificates if that is what a website wants to use. We are always working to improve or extend its validation, issuance, and management processes, including greater use of automation through integrations with certificate lifecycle management (CLM) solutions such as Entrust Certificate Hub, AppViewX, Venafi, and ServiceNow, as well as automation through ACME v2, CMP, SCEP, and other new methods.

We understand that this Google proposal may be causing our customers considerable concern. In accordance with Google’s instructions on its Chrome Root Program Policy, we encourage customers to direct any questions or input regarding the Google proposal to chrome-ro...@google.com; please feel free to share a copy with us at ecs.fe...@entrust.com.
---



3: 2023-07-26: Entrust Cybersecurity Institute Explains: 90-day TLS/SSL Certificate Lifecycle
https://www.youtube.com/watch?v=GcKunPD5SRw
---
0:41-1:48
Today we're using certificates that are generally valid for about a year. Technically, according to the requirements, we can go to 398 days to give you a bit of room so that you can have one day of the year at which you renew your certificates. That is something that is easy to remember, scheduling your agenda as a yearly renewal of those certificates when they need to happen.

When they're moving to shorter-lived certificates, such as the proposed 90 days, this means that you need to do this now four times a year in a window of 90 days.

Moving to 90-day certificates means that you're practically going to renew your certificates probably every 60 days. Meaning that in the end, you have to replace your certificates four to six times a year, depending on what window you allow to renew that certificate.

90-day certificates will have a few benefits. So one is that they will drive automation.


2:51-3:30
That doesn't [mean to] say that there's no problems and that 90-day certificates are actually going to solve this. Crypto agility is important, but that means more than just automating your certificate lifecycle.

For example, if the CA would give you an indication through the automated system that your certificate must be replaced, how does currently the CA know that your system is capable of running a certain algorithm? Have you updated your libraries?

So we still have a long step ahead of us really to make that crypto agility a reality.

4:41-end

We're working with the Internet Engineering Task Force, where Anthos(?) created a proposal for a sort of auto-discovery based on the most used automation, the ACME Auto Certificate Management Environment.

And that would help our customers actually to simplify this mechanism of renewing certificates in different platforms with different systems, without having to reconfigure every individual system to work with the certificate authority and the type of certificates they're dealing with. Together with[sic] very important is one of the risks of automation, is that you're not doing it as a human.

If you do a process, if you follow a step-by-step process that is defined and tested, then you know the outcome, because you're following the steps. But in automation, something might go wrong. And how do you know? You need to make sure that systems are notifying you as someone is watching the notifications.

That's also why in this similar proposal, we've included the mechanism of backup configuration. So if one process would fail, there is another process that can be followed. But other things are still in development. And we will see how that turns out. But the ecosystem seems to be supporting it.
---



4: 2023-07-26: Entrust Cybersecurity Institute Explains: Zero Trust and TLS/SSL Certificate Management
https://www.youtube.com/watch?v=nd0QCvu2F_E
---
1:20-2:18

Well, first I would say we need to adopt automation. But then at the same point, I would point them at the risk of automation. Where processes by humans are easily manageable and can have multiple steps to ensure they flow correctly. Without automation, this could be more challenging.

For example, if you implement automation for your TLS certificates, this means that you need to have a credential for your certificate authority. You need to also have a credential, for example, for your domain name provider, your DNS system. And that's all needed to prove domain control and to make sure that a certificate with the right identity is issued for your endpoints.

But what if these credentials get compromised? Look at the DNS system.
---



5: 2023-08-14: Short-lived Certificates finally approved
https://www.entrust.com/blog/2023/08/short-lived-certificates-finally-approved/
---
After more than 10 years, short-lived TLS certificates are finally permitted by the browsers based on CA/Browser Forum ballot SC-063. Gerv Markham started a short-lived certs discussion in 2014, where he advised he was reviewing the 2012 CA/Browser Forum discussion on the topic. He advised that short-lived certificates was a plank of the Mozilla revocation strategy. There was also a paper prepared in 2012 called Towards Short-Lived Certificates. The paper stated OCSP is as good as dead, so the CAs should issue certificates with a very short lifetime. I suppose no one thought it would take so much time.

Short-lived certificates are designed to help address a certificate revocation issue. Back in 2012, Adam Langley discussed the seat-belt issue, where it works fine, but snaps when you crash. This was based on the fact the browser implements soft-fail revocation checks where the CRL or OCSP response is ignored.
---



6: 2024-03-26: The Path to 90-Day Certificate Validity: Challenges Facing Organizations
https://www.entrust.com/blog/2024/03/the-path-to-90-day-certificate-validity-challenges-facing-organizations/
---
Certificate lifespan is getting shorter

Over the years the cybersecurity industry has undergone notable transformations requiring organizations to implement new best-practice standards, often at a short notice.

In 2020, Apple unilaterally opted for shorter TLS certificate durations, reducing them from three years to 398 days, thereby increasing the burden on certificate management. Subsequently, Apple introduced shorter lifespans for S/MIME certificates at the start of 2022. In the past year, both code signing and S/MIME users faced additional alterations, while Google proposed transitioning to 90-day certificates, a subject we have explored in our latest webinar. Anticipating further changes, particularly with the rise of artificial intelligence (AI) and the looming risk of post-quantum (PQ) computing, organizations must enhance their agility.

Today, TLS/SSL certificates are typically valid for about a year, according to the Certification Authority Browser (CA/B) Forum requirements. This yearly renewal cycle is convenient for organizations to manage and schedule. However, transitioning to shorter-lived certificates, like the proposed 90-day validity period, will require more frequent renewal efforts. With 90-day validity, organizations will need to renew certificates four times every 12 months within that timeframe. In practice, due to the need for buffer time, certificates may need to be renewed every 60 days. Ultimately, this change could lead to replacing certificates more than six times every 12 months, depending on the renewal window chosen.
---

Apologies that some of those got long, I wanted to preserve as much context as possible given how little material we have to work with.

I sincerely ask anyone if they can find any further communication on these topics by Entrust. Their helpdesk has tutorials on specific software setups, but I'm not seeing any actual push for their subscribers to do anything.

It would be very beneficial for Entrust to provide us with any information on what they've been communicating to their customers to promote shorter certificate lifespans and automation.

- Wayne

[Amir Omidi (aaomidi)]

I wanted to also add that I'd like Entrust to address why they don't stop certificate issuances when they find out they're misissuing certificates?

As part of my series on Entrust. In Part 2 I found a concerning issue from Entrust that went unnoticed at the time, which shows a pattern of gross-negligence when it comes to incident response: Entrust: SHA-256 hash algorithm used with ECC P-384 key

In this incident, Entrust discovers that they're misissuing certificates on 2020-06-17. Despite finding out about this incident, they continue to misissue certificates all the way till 2020-06-24: https://crt.sh/?id=2998515551&opt=zlint

There have been a couple of examples of Entrust making this decision in the past, such as: Entrust: Printable String Constraint Failure. Similar to the previous incident, they did not disable issuance on their systems that was capable of misissuing certificates (emphasis mine):

>      Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

        The CA software has a bug which encodes quotation marks in the organization field using PrintableString instead of UTF8String. This software has not been fixed at this time.

And more recently, we saw this behavior in the start of this saga: Entrust: EV TLS Certificate cPSuri missing

[Wayne]

As an advanced warning to Entrust on supplying the report please keep in mind that MDSP has a moderation queue for new members. If the report is to be sent to the mailing list today, then please make sure to use an account that has been pre-approved, or otherwise submit it early enough that a moderator can approve it to arrive on-time.

[Ben Wilson]

All,

I have created Bugzilla Bug#1901270 as an Entrust "meta" bug for gathering all action items that will be included in their report. 

Please don't comment yet in that bug until Entrust has submitted its report and populated the Bugzilla bug with their action items.

Thanks,

Ben

[Bruce Morton]

Please respond to comments you may have on our report or action items here.  We will track our progress against the action items list in Bugzilla under bug 1901270.

[Amir Omidi (aaomidi)]

Can you please explicitly state answers to these following questions - I may have more in the future but these are the immediate ones that come to mind:

Question 1:

Why did section 2.5.1 of your report ignore: https://bugzilla.mozilla.org/show_bug.cgi?id=1890898 - Is this because your legal counsel told you this isn't an incident?

Question 2:

In the report, for the cpsUri incident you state:

> We continued to issue affected EV certificates after reporting the incident because we believed
this to be an error or unintended discrepancy between the CA/Browser Forum’s TLS Baseline
Requirements and the Extended Validation Guidelines. Following feedback from the Bugzilla
community, we fixed the certificates on March 18 and began notifying customers and revoking
affected certificates.

This does not answer my question above which stated:

> I wanted to also add that I'd like Entrust to address why they don't stop certificate issuances when they find out they're misissuing certificates?

A lot of this saga started with that decision to not stop issuance immediately and fix the certificate profile. Beyond that, I'd also like to challenge the "Following feedback from the Bugzilla
community" statement made. You only took action after Ryan Dickson from the Chrome Root Program chimed in. In fact, you actually initially said you don't want to take action: https://bugzilla.mozilla.org/show_bug.cgi?id=1883843#c4, and then once again: https://bugzilla.mozilla.org/show_bug.cgi?id=1883843#c10

When making decisions like this, and making statements like you made in that bug, I expect a CA which (your words):

Entrust has been a certification authority for over two decades, and we have strived to be a
positive influence on development and advancement of our industry’s standards since the
CA/Browser Forum was created in 2005. We recognize that these incidents were unnecessary
and based on our own mistakes or misjudgments – in this we fell short of the high standard we
set for ourselves. We have thoughtfully considered the community’s questions and comments,
and this input is reflected in our plans.

says that about themselves knows that they need to show precedent for making that decision. I was expecting some supporting documents for justifying making that decision to be shown in this report. Can you please confirm if Entrust simply "made it up" when it made that report without consulting any prior incidents, or any supporting documents?

Question 3:

In your action items, you state: "Change organizational structure to enhance support, governance, and resourcing for compliance team". What does this mean? This is a very qualitative statement and I'd like some more details on this. What was it before, how did you identify what needed to change, what changes did you execute?

Question 4:

In your action items, you state: "Expand use of linters pre-issuance for all certificate types" - what certificate types weren't being linted pre-issuance? I believe that Entrust had made a promise to lint all pre-issuances in the past (https://bugzilla.mozilla.org/show_bug.cgi?id=1448986#c7) - is that promise different from this one? If so, how is it different?

Question 5:

 

Has entrust broken this promise they made four years ago? https://bugzilla.mozilla.org/show_bug.cgi?id=1651481#c6 - If not, can you explain how this promise doesn't apply to the recent failed/delay of revocation incidents? If so, can you please help us understand why we should believe you now?

Thanks!

[Wayne]

Okay for my approach I will first note selective parts of this report that jumped out to me:

The highlights section only discusses topics that Entrust should have already been doing, so promising any improvement on that front is the bare minimum.

>2.1.1
>...
>This mis-issuance affected 26,641 EV certificates.
This is factually incorrect, I account for 26,668 impacted certificates and even stated as such in the 20 Incident Revocation Breakdown spreadsheet. I can understand the confusion as Entrust themselves could not find out the correct figure multiple times throughout their own incident, and still cannot.

>Note: During our investigation of this issue, we noted that a subset of 1,975 EV certificates were also issued without the Entrust EV policy identifier (OID), based on our interpretation of the ballot update.
This is also a miscount, presumably due to the original figure being 1963 + 6 certs on a test site that are being double-counted.

>1.
>As of June 7, 2024, all affected certificates related to these incidents have expired or been revoked.
>...
>2.1.1
>CPS Updates: As we updated our EV Certificate profile to resolve bug #1883843, we made two oversights. First, we did not immediately update our CPS to reflect the changes made to the EV certificates on March 18; the issue was fixed in approximately three days. There were 9,045 certificates affected (aside from those already scheduled for revocation as part of bug #1883843), as described in bug #1887753.
>
>Additionally, as we updated the CPS to add the CPS URI qualifier to the EV certificate profile, we inadvertently added this to the OV TLS certificate profile instead. While these certificates were issued in accordance with the TLS Baseline Requirements and our intended certificate profile, there were 6,008 OV certificates issued before the CPS was corrected, as described in bug #1890896.
Those 6000+ certicicates are still pending revocation, with Entrust having previously said they refuse to revoke.

Will they now be revoked?

>2.1.3 Root Cause Analysis
>We misinterpreted the requirement to keep cPSuri in EV certificates, and initially chose not to revoke, believing that the EV Guidelines were in error. Change procedures did not include appropriate review and quality assurance to ensure that updates to TLS Baseline Requirements and EV Guidelines were followed. Additionally, we did not use linters that included the Ballot SC-62v2 changes; we could have used pkilint, which was only in pre-release at that time.
This is an RCA of the technical cause, but not the refusal to listen to anyone telling you that you were wrong. What is not addressed is the continued mis-issuance either.


>2.2.2 Associated Bug List
>Open Date - Bugzilla - Description
>2024-04-03 - 1897630 - EV Locality Errors
Am I missing something or was this opened on 2024-05-19? Note only that the issue in question wasn't officially confirmed internally to Entrust until 2024-05-16. On checking later this is also within the appendix so I presume these dates are not from Bugzilla but from an internal tracker on Entrust's end.

Due to that I propose that Entrust had this incident open internally for over a month before reporting it on Bugzilla as the timeline also suggests.

>2.3.4 Improvement Plan
>...
>Automate CPR form to collect all required information at the outset from the reporter rather than relying solely on email
This goes back to policy issues discussed for years now, see:
https://github.com/mozilla/pkipolicy/issues/98
https://github.com/cabforum/servercert/issues/201
https://bugzilla.mozilla.org/show_bug.cgi?id=1650234

>2.4.1
>One of the reporters of the EV cert issue alerted Entrust that the Problem Reporting Mechanism in its CCADB entry was misaligned with its CPS and contained outdated information (bug 1894111).
To clarify at no time have I reported an EV cert issue to Entrust per their PRM. I am unsure how this got recorded internally on Entrust's end this way. Perhaps they flag anyone commenting on an incident as a reporter?

>2.5.1
>...
>In both incidents, delayed revocation was granted to subscribers who submitted specific requests for exceptions, primarily on the basis of avoiding disruption to critical operations.
This goes against all previous statements that the systems were critical infrastructure and harm would occur.

>2.5.3
>...
>When we addressed delayed revocation in 2020, we made several statements regarding how we would avoid delayed revocation in the future. We have continued to educate our subscribers regarding the need for greater agility and resilience, but we did not make the progress necessary to enable five-day revocation without large-scale disruption to their critical operations. Based on this root cause analysis, we believe the improvement plan below will address the issues identified.
I see no improvement plan that is materially different from prior claims that Entrust has made.

>2.5.4
>...
>We will work with our subscribers to ensure awareness and minimize delayed revocation requests; such requests will be handled only on a case-by-case basis, and only under limited circumstances. We will have this plan in place by the end of June.
I am confused over how your current plan differs from this, or how it will result in revocation occurring within the required timeframes. All that I am seeing is a commitment to offer your subscribers more 'choice' in their revocation timeframe. This is fundamentally misunderstanding the role revocation plays, and that the subscriber is not offered any choices.

>Policy Updates: We ensure that policies are updated and that they are communicated to subscribers. We are considering ways to increase visibility of the CA’s right to revoke certificates on short notice beyond our contract language. We also will add a warning to manual order pages and related emails to ensure subscriber understanding of required timelines.
I was under the impression Entrust did all of this already and have been actively educating subscribers for years as a leader in the field?

>Advancing ACME: We are supporting the ongoing work to automate certificate issuance and management. Entrust experts have authored two IETF drafts around ACME auto discovery, which will help to increase automation adoption by subscribers using public certificates.
Can you make any guarantees that ACME will be a requirement for subscribers going forward, and that they will not be charged extra for using these systems?

>Driving customer adoption of automation: We believe automation is critical to enable ongoing resilience. We have begun campaigns to urge subscribers to adopt automation solutions from Entrust or other providers, including offering our CertHub certificate lifecycle management tool for 12 months at no charge. We are looking at additional ways to drive customer adoption of solutions that enable them to minimize disruption in the event of a five-day or 24-hour revocation.
The first year is free to encourage tie-in answers that question I suppose. It is not on the subscriber to handle 24h or 5d revocation - it is the CA's duty, and so far this report does not understand that.

Now that that is done, let's consider what the report asked for, following the same bulletpoints as the start of this thread:

• I can see the most minimal factors and root causes of initial incidents, and likewise for commonalities. No attempt exists that I can see to address or recognize systemic failures that caused these to occur, not any items address this going forward;
• I see zero reference to any internal policies or protocols used by Entrust, and due to this no evaluation of their internal decision-making;
• I see no detailed timeline of the remediation process, nor an apportionment of delays to root causes;
• I see no mention of historical issues and how they relate to the more recent incidents Entrust has instead focused on.

Now onto the other 3 bullet points:

• I do not personally see clear and concrete steps to address the root causes even as identified, only surface-level reviews that put us back where we started with the technical measures addressed as already required;
• I see no measurable and objective criteria for progress, the majority is internal to Entrust and factors in issues they insist on NDAs and confidentiality over;
• Although if we only consider their action items in a vacuum at least the last one is perhaps addressed?

To that end at most 2 out of 7 elements isn't that bad considering. The strong recommendation on going further than ACME ARI was never addressed either.

I'm sure others will comment but this is my first impressions in a vacuum.

[Watson Ladd]

Dear Bruce,

This report is completely unsatisfactory. It starts by presuming that
the problem is 4 incidents. Entrust is always under an obligation to
explain the root causes of incidents and what it is doing to avoid
them as per the CCADB incident report guidelines. That's not the
reason Ben and the community need this report. Rather it's to go
beyond the incident report to draw broader lessons and to say more to
help us judge Entrust's continued ability to stay in the root store.
The report falls short of what was asked for, in a way that makes me
suspect that Entrust is organizationally incapable of reading a
document, understanding it, and ensuring each of the clearly worded
requests is followed. The implications for being a CA are obvious.

To start Ben specifically asked for an analysis involving the
historical run of issues and a comparison. I don't see that in this
report, at all. The list of incidents only has ones from 2024 listed,
there's no discussion of the two issues specifically listed by Ben in
his email.

Secondly the remedial actions seem to be largely copy pasted from
incident to incident without a lot of explanation. Saying the
organizational structure will be changed to enhance support,
governance and resourcing really doesn't leave us with a lot of
ability to judge success or explain how the changes made (sparse on
details) will lead to improvements. Similarly process weaknesses are
not really discussed in ways that make clear what happened. How can I
use this report if I was a different CA to examine my organization and
see if I can do better? How can we as a community judge the adequacy
of the remedial actions in this report?

Section 2.4 I find mystifying. To my mind there's no inherent
connection between a failure to update public information in a place
it appears, a delay in reconfiguring a responder, and a bug in the CRL
generation process beyond the organizational. These are three separate
functions of rather different complexity. If there's a similarity it's
between the latter two issues where there was a failure to notice a
change in requirements that required action, but that's not what the
report says! Why were these three grouped together, and not others?
What's the common failure here that doesn't exist with the other
incidents?

If this is the best Entrust can do, why should we expect Entrust to be
worthy of inclusion in the future? To be clear, there are CAs that
have come back from profound failures of governance and judgement. But
the first step in that process has been a full and honest accounting
of what their failures have been, in a way that has helped others
understand where the risks are and helps the community understand why
they are trustworthy.

Sincerely,
Watson Ladd

[Wayne]

While Entrust have not provided details on their incident handling and decision-making as requested in this report, a few details have came to light in a reply to an incident today. This is specifically regarding #1886532 the delayed revocation CPSuri certificates.

https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c51

I will do this mailing a list a courtesy by not embedding it all, however I feel that similar details not being included for all of these incidents in this report already is troublesome. This comment shows that Entrust has all of this information available, they just did not feel it worth including despite it being asked for.

[Mike Shaver]

On Sat, 11 May 2024 at 15:04, 'Chris Bailey' via dev-secur...@mozilla.org <dev-secur...@mozilla.org> wrote:

To that end, I want to confirm our intent to provide a full written response to you and the community prior to June 7.

o_o

a full written response to you and the community prior to June 7.

 o_O

prior to June 7

O_______O

Date: Fri, 7 Jun 2024 12:53:10 -0700 (PDT)
From: "'Bruce Morton' via dev-secur...@mozilla.org" <dev-secur...@mozilla.org>
To: "dev-secur...@mozilla.org" <dev-secur...@mozilla.org>
Cc: Ben Wilson <bwi...@mozilla.com>
Subject: Re: Recent Entrust Compliance Incidents

In another context, I would think this to not even be worth joking about, but here it's just the cherry on the top of this whole process.

I have time booked this week to go through the report in more detail (every time I start I turn over another thing that is wrong? it's fractal) but I have to say, now that we've reached the end of this part of the process, that I find Entrust's response--in specific and in general--to be well beneath not only the expectations but indeed the mere *dignity* of the Mozilla root program process, the CA/BF commitments, and the trusted role that Entrust seems to so arrogantly believe cannot be lost.

I am generally known as a pretty charitable person, and in the mists of time when I was responsible for the Mozilla root CA process I very often advocated or outright decided in favour of using incidents as a tool for learning far beyond being a tool for culling underperforming CAs from our root store. Even at the point at which Ben posted the (extremely understated) message beginning this thread, I had hoped that we would see Entrust wake up from its long operational-quality slumber. I had hoped, sincerely, that Entrust would provide plans that were transparent, concrete, thorough, and sufficiently evident of meaningful reflection that the response would be celebrated as an improvement in the health of the WebPKI. It would mean that revenue from the financial disincentive that Entrust puts in place against Subscriber automation (I believe it's called "SUB-PKI-CEG-ACME") might in some small way be put towards strengthening the integrity of the web's security. I was bewildered by the non-responses that kept appearing in the bugs, but honestly I'm a sucker so I remained hopeful. There were VPs involved, Entrust values its security brand so much, their history is so long (I was doing infosec in the Ottawa area in the early 90s)--they were going to come through now that it had been made so abundantly clear that things were structurally broken.

Sadly, I then opened the response posted by Bruce.

When I first read the CPS URI incident, it seemed that Entrust thought that the Mozilla root community wasn't watching them. (To be sure, there had been some evidence in the preceding 4 years that this was the case.)

When the demeanour of Entrust's responses changed immediately after Ryan Dickson of the Chrome Root Program entered the bug, it made me feel that Entrust thought that the Mozilla root program and community didn't matter, and that their commitments to that program were not meaningful.

When the third spokesperson, of increasing seniority, restated Entrust's earnestness and pedigree without any actual concrete, measurable commitments, I started to suspect that Entrust thought that they could just "post through it", as the kids say.

But when I read this report, and especially when I compare it to the exceptionally clear request from Ben in his original message, I can only conclude that Entrust believes that this community and its participants are in fact medically-grade stupid.

I honestly hope that someone there is ashamed of this.

Mike

[Watson Ladd]

On Sat, Jun 8, 2024 at 2:15 PM Mike Shaver <mike....@gmail.com> wrote:
>"It would mean that revenue from the financial disincentive that Entrust puts in place against Subscriber automation (I believe it's called "SUB-PKI-CEG-ACME")"

So for four years, while Entrust told us it was working to get its
subscribers to automate, it was using this as a revenue opportunity
thus continuing manual processes? There is no way to reconcile this
with any sort of commitment here on Entrusts part to getting
subscribers to automate.

Could Mozilla update the root store policy to make clear that
improvements like ACME shouldn't be extra cost items but instead
considered part of the service provided to customers.

[Mike Shaver]

On Sat, Jun 8, 2024 at 6:15 PM Watson Ladd <watso...@gmail.com> wrote:

On Sat, Jun 8, 2024 at 2:15 PM Mike Shaver <mike....@gmail.com> wrote:
>"It would mean that revenue from the financial disincentive that Entrust puts in place against Subscriber automation (I believe it's called "SUB-PKI-CEG-ACME")"

So for four years, while Entrust told us it was working to get its
subscribers to automate, it was using this as a revenue opportunity
thus continuing manual processes? There is no way to reconcile this
with any sort of commitment here on Entrusts part to getting
subscribers to automate.

I find it hard to come to any other interpretation of the facts.

Could Mozilla update the root store policy to make clear that
improvements like ACME shouldn't be extra cost items but instead
considered part of the service provided to customers.

I think that would be an exceedingly reasonable change for Mozilla to make to its root store policy, personally.

Mike

[Mike Shaver]

On Sat, Jun 8, 2024 at 6:29 PM Paul Wouters <pa...@nohats.ca> wrote:

> On Jun 8, 2024, at 18:16, Watson Ladd <watso...@gmail.com> wrote:
>
> 

> Could Mozilla update the root store policy to make clear that
> improvements like ACME shouldn't be extra cost items but instead
> considered part of the service provided to customers.

I don’t have an opinion on this but as someone who at $dayjob has been forced to request non-acme certificates manually, let me assure you that any vendor requiring me to do that quickly gets pulled in the “vendors to migrate away from” list. Any CA preferring manual issuance over automated issuance is going to find itself out of business soon (as are vendors providing web services requiring their customers to send them certs once a year manually while promising to support acme “soon”)

I guess that’s a nice assurance, but what does “soon” mean? July? Are you buying enough certs to swing the economics of a major CA?

The problem right now is Subscribers who *don’t* want to adopt automation, perhaps in part because Entrust would charge them extra for it. They are the excuse being used too frequently for the dereliction of duty.

Mike

[Jeffrey Walton]

On Sat, Jun 8, 2024 at 6:15 PM Watson Ladd <watso...@gmail.com> wrote:
>

I would caution against that. Effectively, Mozilla would be fiddling
with the market. The market should be the one to punish (or reward)
Entrust for the premiums on manual issuance, not Mozilla. When
subscribers get tired of paying too much for the service, the customer
will go elsewhere.

In my mind's eye, there are two things to observe. First is the
CA/Browser Standards ("what we do"), and second is the CA Operating
Procedures ("how we do it"). The Browsers and collective CA's should
focus on the standard (what should be done), and each individual CA
should focus on the implementation (how it is done). The Forum should
not meddle in everyday affairs of a particular CA.

I understand the community wishes to punish Entrust for its chronic
problems. The CA/Browser Forum do not have tools for that, sans
delisting a particular CA. Maybe the CA/Browser Forum needs to adopt
some punishments, like forbidding a CA from issuing OV certificates or
EV certificates for a specified period of time, like a year. Or forbid
the CA from issuing other types of certificates, like S/MIME and code
signing certificates. The year embargo and lost revenue should be
enough of a haircut to get the CA to comply. If a CA continues to defy
the Forum, then delist the CA. There is plenty of competition in the
marketplace, so any particular CA will not be missed.

And remember, there are three parties in the ecosystem. The Browsers
and CA's are only two of them. There are also 5.35 billion relying
parties who use the internet. If the Forum wishes to acknowledge the
interests of the 5.35 billion internet users, then maybe removing
Entrust would be the best course of action. That's because Entrust
only seems to care about itself and its subscribers. It does not seem
to care about the the Forum, the standards produced by the Forum, or
the relying parties. Entrust has lost the trust of the community, and
that is the only commodity that matters to the relying parties.

Jeff

[Mike Shaver]

On Sat, Jun 8, 2024 at 9:48 PM Jeffrey Walton <nolo...@gmail.com> wrote:

I would caution against that. Effectively, Mozilla would be fiddling
with the market. The market should be the one to punish (or reward)
Entrust for the premiums on manual issuance, not Mozilla. When
subscribers get tired of paying too much for the service, the customer
will go elsewhere.

Hey, uh, yeah…Mozilla sort of exists to “fiddle with the market” in ways that it feels protect the web’s users from the direction that The Market might otherwise take. It’s sort of “their thing”.

But that rather jarring dissonance aside, nobody is objecting to premiums on manual issuance. It is precisely the opposite: it is an objection to charging Subscribers *extra* for using *automated* tools that make the web safer (and which indeed should be cheaper for the CA to operate than a manual process, but you know how it is with rent seeking).

The CA’s primary responsibility is to the web’s users, not to its customers. They all know this. It can require that they not always optimize for short-term business outcomes, but if they are not comfortable with that *very* explicit tension, then this is not an appropriate business for them.

In my mind's eye, there are two things to observe. First is the
CA/Browser Standards ("what we do"), and second is the CA Operating
Procedures ("how we do it").

I guess that is a way that these things could have evolved in a parallel universe, but you have perhaps noticed that the BRs already have many directions as to how things must be done. The BRs are in fact growing more such directions over time as it becomes increasingly clear that not all CAs can be trusted to do the things that are best for the health of the WebPKI; see the active discussion about linting practices in the SCWG, for example.

Mike

[Mike Shaver]

Apologies, I somehow managed to send white-on-white HTML from gmail mobile and I honestly have no idea how.

On Sat, Jun 8, 2024 at 9:48 PM Jeffrey Walton <nolo...@gmail.com> wrote:

> I would caution against that. Effectively, Mozilla would be fiddling

> with the market. The market should be the one to punish (or reward)

> Entrust for the premiums on manual issuance, not Mozilla. When

> subscribers get tired of paying too much for the service, the customer

> will go elsewhere.

Hey, uh, yeah…Mozilla sort of exists to “fiddle with the market” in ways that it feels protect the web’s users from the direction that The Market might otherwise take. It’s sort of “their thing”.

But that rather jarring dissonance aside, nobody is objecting to premiums on manual issuance. It is precisely the opposite: it is an objection to charging Subscribers *extra* for using *automated* tools that make the web safer (and which indeed should be cheaper for the CA to operate than a manual process, but you know how it is with rent seeking).

The CA’s primary responsibility is to the web’s users, not to its customers. They all know this. It can require that they not always optimize for short-term business outcomes, but if they are not comfortable with that *very* explicit tension, then this is not an appropriate business for them.

> In my mind's eye, there are two things to observe. First is the

> CA/Browser Standards ("what we do"), and second is the CA Operating

> Procedures ("how we do it").

I guess that is a way that these things could have evolved in a parallel universe, but you have perhaps noticed that the BRs already have many directions as to how things must be done. The BRs are in fact growing more such directions over time as it becomes increasingly clear that not all CAs can be trusted to do the things that are best for the health of the WebPKI; see the active discussion about linting practices in the SCWG, for example.

Mike

[Mike Shaver]

On Sun, Jun 9, 2024 at 3:34 PM Paul Wouters <pa...@nohats.ca> wrote:

On Jun 8, 2024, at 23:53, Mike Shaver <mike....@gmail.com> wrote:

The CA’s primary responsibility is to the web’s users, not to its customers.

That is an interesting view, possibly not shared by its shareholders or the legal framework of the countries they operate in. 

If you have a different view of the BRs to which Entrust and other CAs have committed, or how they conflict in a concrete way with other legal frameworks, then that would be a fine thing to discuss with details in another thread here or perhaps on the CCADB list.

I don’t know what they tell their shareholders, but that’s also not my problem. They don’t have to be in this business, however we got to this situation historically; I think we may well find out that the web can operate just fine without Entrust acting in this capacity at all.

There are many technology businesses which are successful even with the existence of non-profit or similar competition. CAs are not owed a profitable business, especially not at the expense of the integrity of the web’s critical, fragile PKI.

I don’t see how using the DNS and a registrar (instead of a TLS handshake and a root CA) to distribute service identity information fundamentally changes the economics or pressures, but I’m happy to be pointed to something if you think it’s germane to the discussion of how we want CAs to create, or not create, incentives related to automation and certificate agility. Again, perhaps a topic more suited to the CCADB list than to this branch of a discussion of Entrust’s behaviour.

Mike

[Tyrel]

All I can say is... wow. This report seems to treat the recent issues at Entrust as a compliance matter, rather than a matter of trust. Which says it all really. 

I find particularly galling the very clear commitment Entrust made in 2020 to not have any more delayed or failed revocations, and yet here we are in 2024 with a whole slew of them. But rather than then go on to echo the other criticisms, I would like to bring another observation:

Entrust seems to have completely missed the boat with bug #1890898. Even if their revised analysis -- that revocation was not required -- is correct (and I am not a sufficient expert to know), it does not change the fact that for 2 months Entrust believed the certificates were misissued and chose not to revoke for those two months. Which shows blatant disregard for the requirements of the BRs. Especially since this bug then doesn't even make an appearance in section 2.5.1 of this report.

I think Jeffrey Walton hit the nail on the head: "If the Forum wishes to acknowledge the interests of the 5.35 billion internet users, then maybe removing Entrust would be the best course of action." I would go further and suggest that all root programs should look to distrust the Entrust roots (for both TLS and S/MIME purposes) by end of 2024. I just don't see how Entrust, with current management, can plausibly be believed to have any road back to trust.

Tyrel

[Ben Wilson]

All,

This is to acknowledge that we have received Entrust's June 7 Report regarding its non-compliance issues and associated remediation plans. Mozilla will thoroughly review the report and provide comments, requests for clarifications, and verify that the requested items have been and will be addressed in accordance with the request for the Report and the Action Items listed in Bugzilla Bug #1901270.

Our review process will include:

• Assessing the root cause analyses and improvement plans.
• Ensuring that all issues listed in the report and related Bugzilla discussions have been addressed.
• Verifying that the remediation plans align with the expectations and requirements set forth by the Mozilla Root Store Policy and the CA/Browser Forum TLS Baseline Requirements.
• Requesting any necessary clarifications or additional information to ensure comprehensive compliance and future prevention of similar incidents.

We will follow up with specific comments and requests for clarifications as needed.

Thank you for your attention to these important matters.

Ben

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/ebea97dc-1871-4588-93b7-8fd97f50ec0cn%40mozilla.org.

[Mike Shaver]

Does this mean that the window has closed for feedback to Mozilla on the report and its responsiveness to the request?

Will requests for clarification be in this thread, the listed bug, or elsewhere?

Does this mean that Mozilla feels that the action items listed in that bug are sufficiently detailed and concrete that they are appropriate as steps for Entrust to take at this point?

Mike

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaY7Bx30Ck4ZKRP4dKTBfmEqxLKiV6FZPvPN31AbLxWcZA%40mail.gmail.com.

[Ben Wilson]

See below:

On Mon, Jun 10, 2024 at 4:16 PM Mike Shaver <mike....@gmail.com> wrote:

Does this mean that the window has closed for feedback to Mozilla on the report and its responsiveness to the request?

 

No. It just means that we (Mozilla staff) will be reviewing the report and the action items and providing feedback as well.

Will requests for clarification be in this thread, the listed bug, or elsewhere?

Preferably here, but if the requests for clarification are structured in markdown in Bugzilla as replies to Comment 1, then that would be acceptable, too. Otherwise, general comments and critiques should be made here.  For efficiency, it is requested that comments and requests for clarification be collected into as few emails/posts as possible and not be posted in piecemeal fashion.

Thanks,

Ben

[Mike Shaver]

On Mon, Jun 10, 2024 at 7:28 PM Ben Wilson <bwi...@mozilla.com> wrote:

Preferably here, but if the requests for clarification are structured in markdown in Bugzilla as replies to Comment 1, then that would be acceptable, too. Otherwise, general comments and critiques should be made here.

Sorry, I meant: will Mozilla’s requests for clarification from Entrust be posted to mdsp or the bug or etc.?

For efficiency, it is requested that comments and requests for clarification be collected into as few emails/posts as possible and not be posted in piecemeal fashion.

Touché…

Mike

[Ben Wilson]

Hi Mike,

Requests for clarification will be posted here.

Thanks,

Ben

[Ben Wilson]

All,

So far, we have received substantive comments and questions on Entrust’s June 7 Report from Amir, Wayne, and Watson.

Are others planning to submit comments or to request clarification and additional information from Entrust?

Thanks,

Ben

Thanks,

Ben