certificate validation fails for trusted self-signed server certificate

[Cosmin Banu]

I have encountered a scenario with unexpected behavior when trying to connect an SSL client (custom client application) to a server that is configured with a self-signed server certificate (it's not marked as CA, it's a leaf certificate) - more specifically, the server I'm testing against is:
https://self-signed.badssl.com/

I've configured the client by loading the server's certificate in an X509_STORE and then set that store on the SSL_CTX and I've set the verify mode to SSL_VERIFY_PEER.

The error I'm getting is X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT (error code 18: "self-signed certificate").
The documentation for this error suggests even more that this should not happen, since the certificate was loaded into the store:
https://github.com/openssl/openssl/blob/openssl-3.5/doc/man3/X509_STORE_CTX_get_error.pod?plain=1#L193
"The passed certificate is self-signed and the same certificate cannot be found
in the list of trusted certificates."

The introductory documentation doesn't seem to cover this scenario:
https://docs.openssl.org/master/man7/ossl-guide-tls-introduction/#trusted-certificate-store

The OpenSSL version I've tested with is 3.5.0.

Is there something I'm missing? Some other configuration option? Or is this the expected behavior?

[Viktor Dukhovni]

On Wed, Aug 06, 2025 at 09:05:33AM -0700, Cosmin Banu wrote:

> I have encountered a scenario with unexpected behavior when trying to
> connect an SSL client (custom client application) to a server that is
> configured with a self-signed server certificate (it's not marked as CA,
> it's a leaf certificate) - more specifically, the server I'm testing
> against is:
> https://self-signed.badssl.com/
>
> I've configured the client by loading the server's certificate in an
> X509_STORE and then set that store on the SSL_CTX and I've set the verify
> mode to SSL_VERIFY_PEER.
>
> The error I'm getting is X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT (error
> code 18: "self-signed certificate"). The documentation for this error
> suggests even more that this should not happen, since the certificate
> was loaded into the store:
> https://github.com/openssl/openssl/blob/openssl-3.5/doc/man3/X509_STORE_CTX_get_error.pod?plain=1#L193
> "The passed certificate is self-signed and the same certificate cannot
> be found in the list of trusted certificates."

It is perhaps the case that the server certificate loaded into the trust
store is not completely identical with that presented over the wire,
perhaps an older or different certificate with the same subject name is
used on either end.

> Is there something I'm missing? Some other configuration option? Or is this
> the expected behavior?

This works for me (with either the stock OpenSSL 3.2 from Fedora, or
OpenSSL 3.5 as below):

$ PATH=/opt/openssl/3.5/bin:$PATH
$ rm /tmp/ck.pem
$ openssl req -nodes -new -newkey ed25519 -keyout /dev/stdout \
-x509 -days 30 -subj / \
-addext "basicConstraints = critical,CA:false" \
-addext "subjectKeyIdentifier = none" >> /tmp/ck.pem

$ openssl s_server -cert /tmp/ck.pem -accept 127.0.0.1:12345 -quiet -no_ign_eof &

$ sleep 1 | openssl s_client -CAfile /tmp/ck.pem -connect 127.0.0.1:12345 -quiet -no_ign_eof -showcerts
Connecting to 127.0.0.1
Can't use SSL_get_servername
depth=0
verify return:1
DONE

$ openssl x509 -in /tmp/ck.pem -noout -text \
-certopt no_header,no_version,no_serial,no_signame,no_issuer,no_validity,no_subject,no_pubkey,no_sigdump
X509v3 extensions:
X509v3 Authority Key Identifier:
77:E1:C8:B6:23:4A:6C:5B:DE:7F:32:25:16:3C:64:3B:78:32:14:13
X509v3 Basic Constraints: critical
CA:FALSE

The CA bit can be either "TRUE" or "FALSE", the SKID either present or
absent, all the combinations work, provided the exact certificate
presented by the server appears in the trust store.

--
Viktor. 🇺🇦 Слава Україні!

[Cosmin Banu]

That was it, the certificate was different!

I took a snapshot of the certificate a few days ago and it is still valid (not expired), but yesterday they replaced it with a fresh one.

Thank you for the help, much appreciated!