The CT logs are now restored and accepting requests again. The incident impacted the 2023, 2024, and 2025 Cloudflare Nimbus logs. During the impact windows, listed below, Cloudflare was unable to accept and process new log entries. The logs are now fully restored and are catching up. We will send a post-mortem as soon as it’s ready.
Impact windows:
Nimbus 2023 — 2023-11-02 11:44 UTC to 2023-11-03 19:50:00 UTC
Nimbus 2024 — 2023-11-02 11:44 UTC to 2023-11-03 20:00:00 UTC
Nimbus 2025 — 2023-11-02 11:44 UTC to 2023-11-03 18:00:00 UTC
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/80a93305-c019-4040-9ce2-402bc52678b9n%40chromium.org.
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/a06c3b03-45cb-45c2-9eb8-34623faea1b9n%40chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAMa0n6-TqTo2aLFrycp%3DNuVR46FymS-aVPRej0JwrqntHHUVnA%40mail.gmail.com.
Thank you all for the input, we appreciate the feedback from the community. Given the current state of our log, we have decided to take the following steps:
In addition, we will share the root cause analysis, once it’s ready.
Best,
Dina
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/8d82a08f-affd-4ecf-966c-c9fedee68ab2n%40chromium.org.
Hi all,
This email provides an update on Chrome's policy response and perspective on the ongoing Cloudflare Nimbus log outages.
First off, we'd like to acknowledge that this incident is the longest outage from a Chrome-Usable log in many years (possibly ever). While we are grateful to Cloudflare for restoring Nimbus2024, we are aware and sensitive to the risk that this sort of incident causes to the entire CT ecosystem.
However, given the current information available to us, we do not expect to Retire Nimbus2024.
While the significant downtime Nimbus2024 experienced resulted in being unable to verify the inclusion of several thousand certificates with valid SCTs, our understanding is that all certificates for which an SCT has been issued have now been included in the log. While this is a significant violation of the log's MMD, we believe that maintaining the log in the Usable state is in the best interest of our users and the CT ecosystem.
We will continue to audit SCTs encountered by Chrome users for inclusion in Nimbus2024. If that auditing, or other external sources, uncovers evidence of certificates that were never included, we will move quickly to Retire the log.
This incident is ongoing for Nimbus2023, and we hope that log shard can make a similar recovery in the coming days. If so, our response will be similar.
In general, we aim to ensure that replacing logs is as inexpensive as possible, and where there is direct evidence of log corruption, Chrome moves quickly to retire the log and ask that the operator stand up a replacement. However, when we do not have evidence that the log is corrupted, the situation is more complex. Between initial compliance monitoring and ensuring that changes have rolled out to all Chrome clients, standing up a new log takes at least 100 days. Recovering an existing log, even when it has had substantial downtime, can provide substantially better log availability for CAs and other certificate submitters.
Once Nimbus2023 has been recovered, we're asking that Cloudflare provide a postmortem, as we do following all major log incidents, and as they have offered. Postmortems are importantly not about assigning blame, but rather they provide the community to learn from our collective mistakes and thus reduce the likelihood that similar issues happen in the future. In this particular case, we're asking that the postmortem cover:
a root cause analysis for the incident overall,
how Cloudflare was confident that the logs could be fully recovered,
what major factors contributed to the length of the incident, and
what steps Cloudflare is taking to reduce the likelihood of similar outages and long recovery timelines in the future.
Thankfully, by design, Certificate Transparency is robust to the failure of single logs. However, multiple log failures can cause availability issues for certificates. As a reminder, CAs and sites providing SCTs directly who are concerned about the risk of multiple log failures can partly mitigate that risk by providing additional SCTs beyond the minimal set required by user agents.
As always, we're happy to answer any questions you have,
Joe, on behalf of the Chrome CT team
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/5618e5f7-8c33-4e4f-a360-5be5af1d6195n%40chromium.org.
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/865637c4-3530-4154-b95e-ba8fe58caa4a%40mtasv.net.
The question that comes to my mind is given that blown MMDs and uptime
requirements are not (any longer?) disqualifying events for a Qualified log,
should those requirements be removed from the CT Log policy?
...
If blowing the MMD in and of itself introduces no appreciable security risk,
then does there need to be an MMD requirement in the policy? The
counter-factual case, that there *is* a security risk caused solely by a
blown MMD, is somewhat disproven by the fact that the Nimbus logs are not
being removed.
- Matt
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/3b6eba35-80d5-489a-a92b-bdde89c54f39%40mtasv.net.
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/6d311812-a25d-429b-99fb-53e9204ab42b%40mtasv.net.
The Nimbus 2023 log is back up and able to accept new submissions. In addition, all unprocessed entries have been incorporated into the log. We started accepting new entries for the Nimbus 2023 log on 2023-11-22 01:15 UTC. We are continuing to run validation checks on both the Nimbus 2023 and Nimbus 2024 entries and will update the group once the validation checks have been completed.
Best,
Dina
In any event, I'm not questioning the existence of the MMD as a *concept*,
but rather its continued inclusion in the Chrome CT Log policy as a
mandatory adherence, rather than a nice-to-have.
--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/0eefd67d-f2f1-4bbe-a341-ab7bff8f421d%40mtasv.net.