Hi!
I have a custom TLS server-client pair of applications implemented using OpenSSL.
Most of the code has been implemented by snooping at these:
https://wiki.openssl.org/index.php/Simple_TLS_Server
https://wiki.openssl.org/index.php/SSL/TLS_Client
The major difference being that I am using a custom 'verify_callback' as described here:
https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_verify.html.
It connects and I can confirm that my custom verifies callback is working.
However, I have a major problem. If I am using a client with a certificate that is expected to fail the custom verification, the server's 'SSL_Acccept' method fails (as expected) but the client's 'BIO_do_connect' and 'BIO_do_handshake' do not throw any errors and the client thinks it is connected.
After the 'BIO_do_connect' call, this string: 'SSL_state_string_long' returns "SSL negotiation finished successfully".
What can I do on the client-side to verify if the client has not been accepted?
The client code is almost entirely the same as in the TLS_Client above.
If I use the openSSL s_client application, with a certificate that passes the custom verification - it works, if I use it with a certificate that should fail the custom verification - it throws an error (which is what I want my client to do).
Any hints? Is there a 'SSL_Check_if_connected' or 'SSL_check_post_initial_handshake' that I am missing? I can provide more details if required. Thank you!