Unable to retrieve entries from Wyvern and Sphinx

[Andrew Ayer]

https://wyvern.ct.digicert.com/2024h2/ct/v1/get-entries?start=28000&end=28999
and https://sphinx.ct.digicert.com/2024h2/ct/v1/get-entries?start=11000&end=11999
are returning the following error:

Forbidden
backend GetLeavesByRange request failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4276203 vs. 4194304)

Every new Trillian operator faces this until they tweak something.
Hopefully another Trillian operator can chime in with the necessary fix.

Regards,
Andrew

[Phil Porada]

The log operators will want to lower the maximum get-entries request value to something like 512 or 256 instead of the default. What's currently happening is they're hitting the gRPC maximum message size of 4MB. When a request happens, the backing database response gets serialized by the trillian log_server, transmitted to the certificate-transparency-go ctfe over gRPC, and then should be returned to the client, but it's that log_server to ctfe gRPC that's failing.

$ curl -sL https://wyvern.ct.digicert.com/2024h2/ct/v1/get-entries?start=28000\&end=28555 | jq -r '.entries[].leaf_input' | wc -l
556

$ curl -sL https://sphinx.ct.digicert.com/2024h2/ct/v1/get-entries?start=11001\&end=11690 | jq -r '.entries[].leaf_input' | wc -l
690

See these links for some more information:

* https://github.com/google/certificate-transparency-go/blob/a2d98c5789314e464fd63684565e02f2dfd078d4/trillian/ctfe/ct_server/main.go#L65

* https://community.letsencrypt.org/t/enabling-coerced-get-entries/114436

[Andrew Ayer]

Does DigiCert have any updates about this? These are qualified logs
which have given out over 12 million SCTs and monitors can't retrieve
the certificates. IMO, this should be considered log downtime.

Regards,
Andrew

> --
> You received this message because you are subscribed to the Google
> Groups "Certificate Transparency Policy" group. To unsubscribe from
> this group and stop receiving emails from it, send an email to
> ct-policy+...@chromium.org. To view this discussion on the
> web visit
> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20240627134941.7ab06dca58bd3a8f350ef09e%40andrewayer.name.

[Ben Cartwright-Cox]

I don't think it's fair to call it downtime if you can reduce your
range and get a response, for example

https://wyvern.ct.digicert.com/2024h2/ct/v1/get-entries?start=28000&end=28100

works just fine.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20240701123019.ce9e5ffedb0de6b0786a568a%40andrewayer.name.

[Andrew Ayer]

On Mon, 1 Jul 2024 17:33:49 +0100
Ben Cartwright-Cox <b...@benjojo.co.uk> wrote:

> I don't think it's fair to call it downtime if you can reduce your
> range and get a response, for example
>
> https://wyvern.ct.digicert.com/2024h2/ct/v1/get-entries?start=28000&end=28100
>
> works just fine.

RFC 6962 Section 4.6 could not be clearer that a request range of
any size is valid, and that it is the log's responsibility to return
fewer entries if it doesn't want to return the full range:

"If a client requests more than the permitted number of entries, the
log SHALL return the maximum number of entries permissible"

And the Chrome CT Log Policy states that "Outages include, but are not
limited to: [...] HTTP response status codes other than 200".

Regards,
Andrew

[Chuck Blevins]

Hi.

Digicert has completed the changes to the 2024 and 2025 Wyvern and Sphinx logs. Max entries is now set to 256.

Cheers,

Chuck Blevins

 

[Andrew Ayer]

On Mon, 1 Jul 2024 11:26:09 -0700 (PDT)
Chuck Blevins <crb...@gmail.com> wrote:

> Digicert has completed the changes to the 2024 and 2025 Wyvern and
> Sphinx logs. Max entries is now set to 256.

Yup, it's working for me. Thanks!