Logging of final certificates and availability

[Jacob Hoffman-Andrews]

When Let's Encrypt launched SCT embedding, we treated precertificates as sufficient logging. After some feedback from the community (https://twitter.com/j4cob/status/979751081646333952, https://community.letsencrypt.org/t/non-logging-of-final-certificates/58394), we decided to turn on logging of final* certificates. This achieves two things:

 - Makes the embedded SCTs public, for auditing against what is published in logs.
 - Avoids creating a "reservoir" of loggable but unlogged certificates, which could be submitted en masse in an attempt to knock over logs.

However, when we turns on logging of final certificates, we noticed increased error rates from some logs. Since we were concerned about contributing to potential disqualification of a log before the deadline, we disabled final certificate logging for all logs except Argon.

I have a couple of questions for the community:

 - Is the "reservoir" of unlogged certificates a concern, in terms of the ability for a hostile actor to try and knock a log outside of its availability requirements?
 - Are logs allowed to discriminate between certificate types? For instance, during heavy load, could a log prioritize precertificates and plain* certificates over final certificates? Those two submission types are much more likely to be critical for issuance, which in turn is likely to be critical for hosting providers to bring new sites online.

Definitions for this email:

*final certificate: certificate with embedded SCTs
*plain certificate: certificate with neither embedded SCTs nor precertificate poison

[Pierre Phaneuf]

In my capacity as a member of an organisation operating a log, that
reservoir of unlogged certificates is indeed a concern. We do try to fill
our own logs, using certificates found in the wild by the Google search
crawler, for example, but even then, Merkle Town is showing 169M
certificates that could be submitted to Pilot? We have some protection
mechanisms, but it's a worry, with the current log implementations.

Trillian-based logs are faring much better, of course, with even a billion
certificates not being a big deal. :-)

What the logs are allowed to discriminate on is a policy question, of
course, which means it is for others to answer. ;-)

I could imagine a weaker version, where during heavy load, a log might turn
down "final" certificates that have (valid!) SCTs from itself, on the
assumption that they already have some version of it? I'm not sure I would
really go along with that, though, it does seem like it would be best if
both certificates could be logged, in my opinion. We should all be
migrating to Trillian-based logs, basically! ;-)
On Thu, May 17, 2018 at 7:13 PM Jacob Hoffman-Andrews <js...@letsencrypt.org>
wrote:

> --
> You received this message because you are subscribed to the Google Groups
"Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to ct-policy+...@chromium.org.
> To post to this group, send email to ct-p...@chromium.org.
> To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/ct-policy/e238e707-49a2-4797-b50b-12b6862ad85f%40chromium.org
.

[Ryan Sleevi]

On Thu, May 17, 2018 at 3:13 PM, 'Pierre Phaneuf' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

What the logs are allowed to discriminate on is a policy question, of
course, which means it is for others to answer. ;-)

I could imagine a weaker version, where during heavy load, a log might turn
down "final" certificates that have (valid!) SCTs from itself, on the
assumption that they already have some version of it? I'm not sure I would
really go along with that, though, it does seem like it would be best if
both certificates could be logged, in my opinion. We should all be
migrating to Trillian-based logs, basically! ;-)

I think the migration to Trillian logs is bad advice - we absolutely shouldn't encourage a Log Implementation monoculture - but perhaps you meant to the extent of logs that take architectural scale designs to the core.

 

So, I've offered several suggestions to log operators to make sure they're thinking about this.

1) Overall rate limiting for all submission issuance

2) Rate limiting per-root to ensure an equitable balance under load, but allow consuming available resources when they're available and otherwise unused.

3) Rate limiting by-intermediate underneath a given root. That is, the root might be 100 certs/second, with an equitable distribution of buckets amongst intermediates under that root.

The assumption here is that the 'cost' in load is the process of actually issuing the SCT, which involves reliably ensuring stable storage for the certificate prior to signing a committment to integrate. The above suggestions allow fanning out the ingestion front-ends before passing on to the 'backend' that the cert is ready to be logged. It presumes that the resource cost in #1-#3 is one of CPU and network - which can easily scale horizontally - before going to the point of requiring consistency, which is understandably trickier to scale.

[Devon O'Brien]

On Thursday, May 17, 2018 at 11:13:01 AM UTC-7, Jacob Hoffman-Andrews wrote:

When Let's Encrypt launched SCT embedding, we treated precertificates as sufficient logging. After some feedback from the community (https://twitter.com/j4cob/status/979751081646333952, https://community.letsencrypt.org/t/non-logging-of-final-certificates/58394), we decided to turn on logging of final* certificates. This achieves two things:

 - Makes the embedded SCTs public, for auditing against what is published in logs.
 - Avoids creating a "reservoir" of loggable but unlogged certificates, which could be submitted en masse in an attempt to knock over logs.

However, when we turns on logging of final certificates, we noticed increased error rates from some logs. Since we were concerned about contributing to potential disqualification of a log before the deadline, we disabled final certificate logging for all logs except Argon.

I have a couple of questions for the community:

 - Is the "reservoir" of unlogged certificates a concern, in terms of the ability for a hostile actor to try and knock a log outside of its availability requirements?
 - Are logs allowed to discriminate between certificate types? For instance, during heavy load, could a log prioritize precertificates and plain* certificates over final certificates? Those two submission types are much more likely to be critical for issuance, which in turn is likely to be critical for hosting providers to bring new sites online.

Just a note on the comparison of value between pre-certs and final certs. It's likely, but not necessarily the case that logging a final certificate is less valuable than a pre-certificate. Past advice for site operators for surprise.product.company.com has been to obtain an unlogged certificate to install and perform QA, perhaps disabling CT for user agents that enforce it, and then logging shortly before going live and delivering SCTs via TLS/OCSP. This is admittedly an edge case, but CT Logs cutting off final cert issuance due to prioritization could adversely impact legitimate non-embedded SCT delivery situations.

[Ryan Sleevi]

I think that was covered by below - they would be 'plain' certs (same priority as precerts) rather than 'final' certs

 

 

Definitions for this email:

*final certificate: certificate with embedded SCTs
*plain certificate: certificate with neither embedded SCTs nor precertificate poison

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/82e8ffc9-eeef-4a48-a978-caae71c19524%40chromium.org.

[Devon O'Brien]

*final certificate: certificate with embedded SCTs
*plain certificate: certificate with neither embedded SCTs nor precertificate poison

While you've defined final certificates as one with embedded SCTs, it's not clear to me whether parsing out SCTs to determine presence in my example would be remarkably more efficient than blocking all final certs. Regardless, a naive block on final certs could still impact the use case I mentioned.