2020.09.24 period of increased error rates from Google logs

[Josh Aas]

This morning I was looking at the internal service level objective metrics for Let's Encrypt API endpoints and noticed a slight drop overnight. An engineer did a quick log search and turned up approximately 111 submission failures to Google's logs. That is, there were 111 instances where we failed submission to both Argon2020 and Xenon2020. During that period we were still seeing mostly successes from those two logs. We had a successful submission rate of 12 qps for each log, and a failed submission rate about 1 qps for each log.

A community member, Jonathan Rudenberg, pointed out that Google had some general service disruptions affecting services like Gmail from approximately 6pm to 6:30pm Pacific the previous evening. Our engineer then confirmed that the submission failures all happened during that time frame. This leads me to believe that whatever was affecting Google services like Gmail was likely also affecting Google's CT logs.

Cert Spotter confirms a 1 day error rate of 2.79% for Argon vs 0.04% over the past 90 days.

https://sslmate.com/certspotter/stats

--
Josh Aas
Executive Director
Internet Security Research Group
Let's Encrypt: A Free, Automated, and Open CA

[Josh Aas]

Here is more information about the Google outage I was referring to:

https://twitter.com/uhoelzle/status/1309313556328841216?s=20

[Andrew R. Whalley]

Hi Josh,

Thanks for the details of what you saw; they've been passed on to the relevant team.

Cheers!

Andrew

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/767ec15f-7b15-4f1a-b102-7b91a2571a54n%40chromium.org.