Introducing OCSP Watch to Monitor OCSP Responder Reliability

[Andrew Ayer]

Recently, we've seen several CA incidents related to the failure to
provide OCSP responses, such as not operating OCSP services for abandoned
certificate orders <https://bugzilla.mozilla.org/show_bug.cgi?id=1758027>,
or publishing OCSP responses with lengthy delays
<https://bugzilla.mozilla.org/show_bug.cgi?id=1758372>.

To smoke out these issues, I've created OCSP Watch, which continuously
audits a sample of unexpired certificates found in Certificate
Transparency logs to make sure CAs are correctly operating OCSP
services. OCSP Watch has identified over 1,000 certificates issued by
20 distinct CAs for which the CA is not providing a valid OCSP response.

The list of certificates can be found here:

https://sslmate.com/labs/ocsp_watch/

OCSP responses are considered valid if the responder returns a response
within 10 seconds that can be successfully parsed by Go's
golang.org/x/crypto/ocsp package, and has a status of Good or Revoked.

CAs should examine the above list and file an Incident Report if necessary.

Regards,
Andrew

[Rob Stradling]

Hi Andrew.  Thanks for creating this!  One initial comment...

I've started looking at the Sectigo OCSP responses that you've flagged, along with the corresponding OCSP requests that you sent.  There seems to be a pattern for the requests that led to "malformed" responses: the base64 encodings of those requests (the ones I've looked at anyway) all seem to contain one or more occurrences of two consecutive forward-slashes.  I can only reproduce a "malformed" response if I don't URL-encode those occurrences of "//" in an OCSP GET request.

Quoting Corey from https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/cMegyySSqhM/m/G7s5tFR4BAAJ:

“/” is noticeably absent from the pchar production rule (which makes sense, as it’s used as the path segment delimiter), so it must be percent-encoded."

So I believe that "malformed" is actually the correct response here.  See also https://crt.sh/ocsp-responders, which amongst other things does a test for "GET request containing multiple forward-slashes".

Can you confirm whether or not you're leaving "//" un-URL-encoded?

Also, is your code for ocspwatch publicly available?

---

From: dev-secur...@mozilla.org <dev-secur...@mozilla.org> on behalf of Andrew Ayer <ag...@andrewayer.name>
Sent: 22 March 2022 15:14
To: dev-secur...@mozilla.org <dev-secur...@mozilla.org>
Subject: Introducing OCSP Watch to Monitor OCSP Responder Reliability

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Recently, we've seen several CA incidents related to the failure to
provide OCSP responses, such as not operating OCSP services for abandoned

certificate orders <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.mozilla.org%2Fshow_bug.cgi%3Fid%3D1758027&amp;data=04%7C01%7Crob%40sectigo.com%7C6d83072607f349d06e2d08da0c16bcd6%7C0e9c48946caa465d96604b6968b49fb7%7C0%7C0%7C637835589067323313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=H%2FHYJFcAIsa5PJFlchBwuwqXYVWmHEQI7ZwoBjDj%2Fow%3D&amp;reserved=0>,

or publishing OCSP responses with lengthy delays

<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.mozilla.org%2Fshow_bug.cgi%3Fid%3D1758372&amp;data=04%7C01%7Crob%40sectigo.com%7C6d83072607f349d06e2d08da0c16bcd6%7C0e9c48946caa465d96604b6968b49fb7%7C0%7C0%7C637835589067323313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=OHNa5DARA0H1pkW9aI%2FHxFBR7Rsf0riBxhjSaFwcX2E%3D&amp;reserved=0>.

To smoke out these issues, I've created OCSP Watch, which continuously
audits a sample of unexpired certificates found in Certificate
Transparency logs to make sure CAs are correctly operating OCSP
services.  OCSP Watch has identified over 1,000 certificates issued by
20 distinct CAs for which the CA is not providing a valid OCSP response.

The list of certificates can be found here:

https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsslmate.com%2Flabs%2Focsp_watch%2F&amp;data=04%7C01%7Crob%40sectigo.com%7C6d83072607f349d06e2d08da0c16bcd6%7C0e9c48946caa465d96604b6968b49fb7%7C0%7C0%7C637835589067323313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=ARSrjZ8C6WPePPmGM757SZzi9VS0xLCuNn6fFtdz2l8%3D&amp;reserved=0

OCSP responses are considered valid if the responder returns a response
within 10 seconds that can be successfully parsed by Go's
golang.org/x/crypto/ocsp package, and has a status of Good or Revoked.

CAs should examine the above list and file an Incident Report if necessary.

Regards,
Andrew

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fa%2Fmozilla.org%2Fd%2Fmsgid%2Fdev-security-policy%2F20220322111454.dd41936a93e1b576818510d9%2540andrewayer.name&amp;data=04%7C01%7Crob%40sectigo.com%7C6d83072607f349d06e2d08da0c16bcd6%7C0e9c48946caa465d96604b6968b49fb7%7C0%7C0%7C637835589067323313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=IJyqkVyJl2e7vlp42OzzIJQ3z7QhpiFyO%2BoaPE6O4qY%3D&amp;reserved=0.

M

[Andrew Ayer]

Hi Rob,

I'm sending POSTs rather than GETs, so this shouldn't be an issue.

The code for OCSP Watch isn't publicly available, but I'll look into
pulling out the core bits into a library that can be open sourced.

Note that crt.sh is also getting a "malformed" response for at least some
of these certs, e.g. https://crt.sh/?id=6393049625&opt=ocsp

Regards,
Andrew

> https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/MW4PR17MB47294004773DF94FF7CE8B07AA179%40MW4PR17MB4729.namprd17.prod.outlook.com.

[Adriano Santoni]

Thanks for bringing this to our attention. We started investigating and will inform the community of our findings as soon as we have a clear enough picture.

For now we can say that the irregular OSCP responses reported by your OCSP Watch concern 3 pre-certificates.

Adriano

ACTALIS S.p.A.

[Andrew Ayer]

The code that OCSP Watch uses to evaluate a certificate's OCSP
responder is now public: https://github.com/SSLMate/ocsputil

Regards,
Andrew

[Andrew Ayer]

On Thu, 24 Mar 2022 05:46:37 -0700 (PDT)
Viktor Varga <viktor...@gmail.com> wrote:

> We found out, that the OCSP Checker gives false positives on OCSP
> responder certificates.
> The responders have OCSP No Check extension, so the responders are
> not giving valid answers on requests.
> All the Microsec results are false positives.

Hi Viktor,

Thanks for finding this and letting me know. I'm now excluding these
certificates from OCSP Watch. I also found and removed 2 certificates
issued by Microsoft which were false positives for the same reason.

Regards,
Andrew

[Viktor Varga]

Dear Andrew,

Thank you for the quick fix.
Kind regards,

Viktor

[Adrian Müller]

Dear Andrew,
Dear all,

Thank you for providing this work regarding OCSP responses and for sharing the detailed list of concerned certificates. We have analyzed SwissSign’s certificates on the list and herewith give a wrap-up of our insights.

SwissSign’s 10 certificates on the list provided under https://sslmate.com/labs/ocsp_watch/ all share the same specifics:

They are precertificates where no final certificates with the according serial number have been issued, the OCSP response for these certificates is «unknown» and they were issued before July 2021.

In September 2021 SwissSign changed the handling of precertificates to not only be compliant with BRG 4.9.10 (which states that a definitive OCSP response is optional for precertificates) but also to follow the recommended practice detailed in “CA/Required or Recommended Practices” (https://wiki.mozilla.org/CA/Required_or_Recommended_Practices).

Because of this constellation (MAY clause in BRG and Mozilla recommended practice) we assumed that it was acceptable to simply let the remaining precertificates in the status “unknown” where no final certificate was issued.

 
Best regards

Adrian

 

Adrian Mueller

SwissSign AG

[Adriano Santoni]

Based on our y2021 logs, we found that the incorrect OCSP responses for those 3 pre-certificates was caused by an undetected and isolated DB problem during the insert phase. That specific issue was addressed by an update we deployed in August 2021, so the same situation cannot happen any more. For those 3 pre-certificates we have remedied the situation by implementing a specific procedure to re-insert the missing pre-certificates into our DB.

Thanks again, Andrew, for reporting the anomaly.

Adriano

ACTALIS S.p.A.

[Vijay Kumar]

Hi Andrew,

Thanks for this work. We had a check on the counts coming under our name (eMudhra). The problem indicated for all certs are "OCSP response has invalid content type application/x-x509-ca-cert".

I believe this is an acceptable response and there is no problem. The OCSP response are signed via dedicated responder cert (not the CA), and hence it contains this cert data. Else the OCSP verification fails.

Appreciate if you/someone can suggest if I'm missing something here. 

Regards,

Vijay

[Corey Bonnell]

Hi Vijay,

RFC 6960, Appendix A.2 says:

 

“An HTTP-based OCSP response is composed of the appropriate HTTP

   headers, followed by the binary value of the DER encoding of the

   OCSPResponse.  The Content-Type header has the value

   "application/ocsp-response".”

 

I believe this is clear guidance that the Content-Type must be “application/ocsp-response” regardless of whether delegated responder certificates are used.

 

Thanks,

Corey

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/d039e4b0-e009-46fd-9402-a9e107a8fb75n%40mozilla.org.

[Ryan Sleevi]

On Wed, Apr 6, 2022 at 12:09 AM 'Vijay Kumar' via dev-secur...@mozilla.org <dev-secur...@mozilla.org> wrote:

I believe this is an acceptable response and there is no problem.

Can you explain why you believe this is? What standards or resources support this interpretation?

I realize that's a very direct/blunt way of asking a question, but mostly, it's useful to include the evidence/thought process with the "I believe" statements. It's totally fine to be wrong - no one is going to get everything right all of the time - but the more explanation about how/why conclusions were reached, the more we can do to improve the guidance or figure out where processes are failing.

 

The OCSP response are signed via dedicated responder cert (not the CA), and hence it contains this cert data. Else the OCSP verification fails.

Containing a certificate is different than indicating that mimetype. The mimetype indicates the URL contains a particular format.

Looking at https://www.iana.org/assignments/media-types/media-types.xhtml , we can see application/x-x509-ca-cert was registered by https://www.rfc-editor.org/rfc/rfc8894.html , which https://www.rfc-editor.org/rfc/rfc8894.html#name-registration-of-the-applica indicates this is a legacy synonym to application/pkix-cert, and the expectation is a DER certificate.

Meanwhile, https://datatracker.ietf.org/doc/html/rfc6960#appendix-C.2 is quite clear that the expected MIME type for an OCSP response is application/ocsp-response , and is required by https://datatracker.ietf.org/doc/html/rfc6960#appendix-A.2

[Vijay Kumar]

Thanks Paul, Corey, Ryan.

The responses analysed by us led to believe that 'OCSP response containing certificate value in the ocsp response is causing the issue'. We were wondering that it is right, as RFC 6960 permits OPTIONAL certificate value in the response. Some of our sample tests gave us proper MIME Type response (different certs). Hence, was trying to ask if there is something in the 'OCSP watch' code, when a different responder certificate is used.

Sorry about the confusion here. It looks there was problem in the OCSP URL, which is being worked now. It was giving a different content-type response impacting some of the certificate issuances recently. The incident details would be published separately.

Sorry about the blunt way of asking question. As we know, the non-native English speakers usually lead to drafting the questions in improper manner. But this is a great advise and it would have been helpful if I had elaborated my understanding / belief in first instance itself. Thanks.

Vijay.

[Paul van Brouwershaven]

Hi Vijay,

You might want to check RFC 6960 appendix A - OCSP over HTTP:

An HTTP-based OCSP response is composed of the appropriate HTTP
headers, followed by the binary value of the DER encoding of the
OCSPResponse. 

The Content-Type header has the value
"application/ocsp-response". The Content-Length header SHOULD
specify the length of the response. Other HTTP headers MAY be
present and MAY be ignored if not understood by the requestor.

• https://datatracker.ietf.org/doc/html/rfc6960#appendix-A.2
• https://datatracker.ietf.org/doc/html/rfc6960#appendix-C

Paul