CT Logs rejecting certificates with large key sizes

[Devon O'Brien]

Hello folks,

We've been seeing some discussion on Twitter about CT Logs rejecting leaf certificate submissions due to them containing a very large key size (32k+). While we're still looking into whether (and, if so, why) these rejections are being performed for leaf certificates in major CT Log implementations, I wanted to chime in with the Chrome CT Policy perspective on this.

Since certificate submissions do not require a CT Log to perform any cryptographic operations over the leaf certificate key, there is no obvious need for CT Logs to treat them as anything other than opaque objects like any other field (barring the signature, of course). 

Regardless of whether these certificates are reasonable for practical use, CT's primary goal is to ensure that certificate and pre-certificates issued from publicly-trusted CAs are made available for monitoring, and we are inclined to treat such rejections as a violation of the Chrome CT Log Policy.

To that end, I have a couple questions for the community:

1. CAs and folks submitting certificates to CT Logs: Have you encountered unexpected certificate logging rejections from Qualified/Usable CT Logs? If so, can you explain the circumstances?
2. CT Log Operators: Are there any operational reasons to reject certificate logging submissions due to size, key size, or any other reason not covered by the existing Crome CT Log Policy?

-Devon

[Burton]

Hello Devon,

I was the person who started the Twitter conversation then deleted the tweets to post on here at Ryan's suggestion.

On the 19th of January, I emailed Mads asking if it's possible for Buypass to issue a domain validated certificate that contained an RSA 32,768 key and attached the CSR. Previously, Mads provided a domain validated certificate that contained an RSA 16,384 key that was issued earlier. The certificate: https://crt.sh/?id=3949938720.  

Mads emailed back later that day saying,

"Looks like we have some issues with issuing certificates for such large key sizes, we must do some further investigations to find out why."

Today on the 22nd of January, Mads emailed back saying,

"We have done some investigations and we see that the certificate issuance process fails when we try to register the pre-certificate in a CT-log. We do not know the exact reason, but it could be some limitations in certificate/key size supported by CT-logs or CT-log operator."

Thank you

[Burton]

Posting on behalf of Mads.

Hi

 

I am very sorry I gave the impression that there is an issue with submitting leaf certificates with very large key sizes into CT logs.

 

The answer I gave Burton was unfortunately not based on a very precise diagnosis of the situation. We observed that the registration of the pre-certificate failed, but we had no clear evidence that this was due to the certificate being rejected by the CT logs.

 

Issuing certificates for such large key sizes is a very rare case and this was our first attempt to issue a certificate for such a large key. When this failed, we were pretty sure that this had to do with the key size and we did not put much effort into identifying the exact cause.

 

However, after doing some more investigations now we see that the failure is caused by an error in a Buypass infrastructure component between our internal servers and the external CT logs.

 

Regards

Mads

 

On Mon, Jan 25, 2021 at 1:52 PM Mads Egil Henriksveen <Mads.Hen...@buypass.no> wrote:

Hi

 

I am very sorry I gave the impression that there is an issue with submitting leaf certificates with very large key sizes into CT logs.

 

The answer I gave Burton was unfortunately not based on a very precise diagnosis of the situation. We observed that the registration of the pre-certificate failed, but we had no clear evidence that this was due to the certificate being rejected by the CT logs.

 

Issuing certificates for such large key sizes is a very rare case and this was our first attempt to issue a certificate for such a large key. When this failed, we were pretty sure that this had to do with the key size and we did not put much effort into identifying the exact cause.

 

However, after doing some more investigations now we see that the failure is caused by an error in a Buypass infrastructure component between our internal servers and the external CT logs.

 

Regards

Mads

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/266bef4a-e4a3-49b9-8efc-8c467397e90an%40chromium.org.

[Devon O'Brien]

Thanks so much for providing more information on where in the issuance pipeline the error occurred. I'm glad to hear that (to the best of our knowledge, anyways) we don't have any outstanding issues of CT Logs improperly rejecting logging submissions.

Beyond this particular instance, we're always very interested in hearing about issues related to unexpected rejection of certificate logging submissions, so feel free to consider the two questions I posed previously as long-standing ones that we're happy to discuss as issues arise.

-Devon

[Jacob Hoffman-Andrews]

I meant to reply on this earlier, and was reminded by the recent discussion about the DigiCert logs.

Let's Encrypt's logs used to have a client max body size of 50kB. After reading this thread, we bumped it up to 23MB. We verified via our metrics that we had not rejected any certificates in the last 90 days due to the size limit.

We justify the 23MB limit based on the TLS spec: https://tools.ietf.org/html/rfc8446#section-4.4.2:

      struct {
          opaque certificate_request_context<0..2^8-1>;
          CertificateEntry certificate_list<0..2^24-1>;
      } Certificate;

In other words, the Certificate message in TLS (which includes the whole chain) can be no larger than 2^24 bytes (16.7 MB). Multiply that by 4/3 for base64 encoding, round up, and you get 23MB. Any certificate that could be used in a TLS handshake should be uploadable given those constraints Caveat: one can imagine a 16.7 MB end-entity certificate, served with no chain, but relying on AIA fetching or cached intermediates. Submitting this to CT via add-chain would require including a chain, which might push it over the limit.

I think Chrome CT policy should set a submission size that logs are required to support, and allow logs to reject submissions above that size. Operationally, there's always a limit, whether it's memory, disk, or network timeouts. It's better to be explicit about what it is. Also, setting explicit limits is useful in constraining the resources that an untrusted client can consume. CT submissions must be read in full and parsed at multiple layers (JSON, base64, x509) before the log can reject based on the contents not being a real certificate, or not being signed by a trusted root.

DigiCert mentioned they have Nginx configured to store proxy_pass responses to disk if they exceed a certain size. We have a similar configuration on the client side: If a client sends a POST body larger than a certain size, we'll store it on disk until the whole thing is read. This is fine, but it adds complexity, and means we have a rarely-exercised path that could cause availability issues at some point.

> CT's primary goal is to ensure that certificate and pre-certificates issued from publicly-trusted CAs are made available for monitoring

In support of this goal, if Chrome allows CT logs to reject very large submissions, it would make sense to also impose a maximum size of certificate that Chrome accepts in handshakes. This would be good for users: If there is a 16MB certificate out there, it is undoubtedly generating a bad user experience due to slow site loads.

[Ryan Sleevi]

Just to make sure this isn't missed: https://github.com/chromium/ct-policy/issues/31 to track this suggestion :)

Folks should feel free to open issues against the policy with suggestions or improvements, as well as file issues :) Certainly, discussion on list helps make sure there's consensus, so it's definitely our preferred venue for pre-flighting, but I also don't want folks to feel their very useful suggestions get ignored :)